Archiv der Kategorie: Java, C and more

Home, Java, C and more

MapReduce for Education

MapReduce represents a pattern that had a huge impact on the data analysis and big data community.  Apache Hadoop allows to scatter and scale data processing with the number of nodes and cores.

One of the many corner points in this full framework is that code is shipped and executed on-site where the data resides. Next, only a pre-processed transformed version (map) of the data is then shuffled and sorted to the aggregators on different executors via the network.

MapReduce is hard to use on its own, so it usually is deployed with
Apache Hadoop or Apache Spark. To play around with it without either one of those large frameworks, I created one in Python – MapReduceSlim. It emulates all core features of the MapReduce. It has one difference, it loads each line of the files separately into the map function. In the case of Apache Hadoop, it would be block-wise. This provides a nice solution to understand the behavior and the pattern of MapReduce and how to implement a mapper and reducer.

Classic WordCount Example

Mapper function

# Hint: in MapReduce with Hadoop Streaming the 
# input comes from standard input STDIN
def wc_mapper(key: str, values: str):
    # remove leading and trailing whitespaces
    line = values.strip()
    # split the line into words
    words = line.split()
    for word in words:
        # write the results to standard 
        # output STDOUT
        yield word, 1

Reducer function

def wc_reducer(key: str, values: list):
    current_count = 0
    word = key
    for value in values:
        current_count += value
    yield word, current_count

Finally, call the function with the MapReduceSlim framework

# Import the slim framework
from map_reduce_slim import MapReduceSlim, wc_mapper, wc_reduce

### One input file version
# Read the content from one file and use the 
# content as input for the run.
MapReduceSlim('davinci.txt', 'davinci_wc_result_one_file.txt', wc_mapper, wc_reducer)

### Directory input version
# Read all files in the given directory and 
# use the content as input for the run.
MapReduceSlim('davinci_split', 'davinci_wc_result_multiple_file.txt', wc_mapper, wc_reducer)

Further information @ Github: https://github.com/2er0/MapReduceSlim

 

Home, Java, C and more, Linux, Sticked!

R (rlang) and Plots everywhere

Data science and Jupyter notebook can sometimes get exhausting. What about debugging, version control, code reviewing and so on. Coming from a Software Engineering background it‘s like losing 50% of the stuff you were used to.

To mitigate those problems I recently partially switched from Python to R with many improvements. For local Python coding, JetBrains PyCharm is my tool of choice and Jupyter notebooks for remote coding. With R it is RStudio Desktop and for remote, there is RStudio Server, which is almost like the desktop version within a browser. This allows one to develop and analyze data from any device with a browser.

R-Studio Server with running R-Notebook sample

R (rlang) and Plots everywhere weiterlesen →

Home, Java, C and more, Linux, Network, Sticked!

Docker a nice tool for developers

Docker is the world leading software containerization platform. I tried using GitLab as versioncontrol system and Jenkings as continuos integration system but the system turned out as not completly useful.

In that way a colleague told me about a system he wants to create based on Docker.

What we want:

  • a git system under ouer control
  • a continuos integration that is flexible and customizable
  • every system available per SSL connection

Git Service -> GoGs – Go Git Service

It’s a simple self-hosted Git servise.

  • easy to install, cross-platform, lightweight and OpenSourse

It contains everything importent to develop something in collaboration.

https://gogs.io/

Continuos Integration -> drone

It’s a continuous integration platform build on container technology. Every build run will be triggerd by a push to a repository if it’s linked to drone.

  • flexible and customizable: by setting up a config file you tell drone what is to do

https://github.com/drone/drone

SSL -> NGINX as reverse proxy

It’s a fine powerfull tool and a nice reverse proxy. With it we are able to provide the GoGs and the drone to the internet more secure and with SSL encryted.

We will colleced the SSL/TLS certificates from a Let’s Encryt service.

https://www.nginx.com/ https://letsencrypt.org/

Docker a nice tool for developers weiterlesen →

Home, Java, C and more, Linux, Sticked!

YouTube Audio Player

screen

For one year I have tried to create my own YouTube Audio Player. I wanted something that can play my playlists and favorite music from YouTube, but I only wanted the sound – no video. In that case I usually used a separate browser-window or -tab and that was sometimes very annoying.

Download all OS

So this player is build on top of NodeJS and Electron and uses the Google API for all YouTube requests. The code and the structure is available on GitHub.

Home, Java, C and more, Sticked!

Angular 2, Electron, NodeJS

Currently I’m working on a private project with Angular 2 (v.2.0.0-beta.0 | TypeScript), Electron (Atom Shell) and NodeJS.

I had some issues with Angular 2 and webpack modular bundler.
In my system I have to transcode all the TypeScript code with all the nice features to normal JavaScript-code (ES5).

first issue
  • It took me many hours to get the InterProcess Comunication from the Electron to the NodeJS running. The Problem is that the IPC-module is a native Electron-module and webpack is trying to find it in the node_modules folder.
    StackOverflow issue

Solution: webpack-target-electron-renderer
You have to change/update your webpack config. After that webpack will ignore or find the native Electron-modules.

second issue
  • In Angular 2 you can write components, services, …. and more. I wanted to use a service to get data from one component to another but I never got the data exchanged between them.

Solution: In my case I used the Componenttag “providers” to create an instance of the declared service but every time you write “providers” with a service in it, you are creating a new instance of the service and you will never get data from one component to another. You start your app with that

bootstrap(App, ['services']);

since then you have an instance of the provided services. To use them, you only have to import the service and to write in the somponent-constructor

constructor( %varname% : %servicename% )

 

conclusion:

The combination of Angular 2 (TypeScript), Electron (Atom Shell) and NodeJS is very nice. With that components you are able to build a client application rapidly. My next “nice to have” is to integrate a CSS-preprocessor (SASS, LASS, …) in my projects to write CSS-code in the ways of a programmer.

Java, C and more

Schach 8 Damen Problem mit Java

Ich habe mich in letzter Zeit ein wenig mit der Programmiersprache Java beschäftigt. Das Ergebnis hiervon ist, dass ich das “8 Damen Problem” in Schach mit Java gelöst habe.

  • Was ist das “8 Damen Problem” in Schach?
    • Dies bedeutet, dass man 8 Damen auf einem Schachbrett so platzieren soll, dass keine Dame eine andere von Anfang an schlagen kann. Das Problem ist aber, dass die Dame folgendes darf: “Die Dame darf auf jedes freie Feld in jeder Richtung linear und diagonal ziehen, ohne jedoch über andere Figuren zu springen und vereint somit die Wirkung sowohl eines Turms als auch eines Läufers in sich.” Schach 8 Damen Problem mit Java weiterlesen →