17 March 2017

Netflix' data warehouse is based on Hadoop


I'm listening to Rüdiger from Microsoft. He explains what can be provided by Azure, HDInsight, a Big Data platform based on Hadoop.

This brings me to a blog entry I wanted to write for a while.

There are a couple of interesting articles on the Netflix blog about their use of Hadoop for their data scientists for analysis, and as a data warehouse platform:
As you can see, they don't use relational databases as a backend for their data warehouse.

Would a Hadoop platform be a simple, cost effective data warehouse option?

04 March 2017

The fascinating world of the Spring Boot logging pattern

I was recently adding Spring Cloud Sleuth to my Spring Boot applications, and I was wondering how I can make sure that my logging pattern is used but I will also have the sleuth elements in there. I was surprised that this seemed to work like magic, out of the box, nothing big to change for me, except to pull the default logigng pattern from the defaults.xml file, put it into my application.yml file in a nice way, done.

But as a curious person, I asked myself how this was possible. I found the explanation on https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html, under chapter 26.5. Heureka - Spring Cloud Sleuth uses the property "logging.pattern.level" to inject its own elements inot the logging pattern by replacing the level item. Sneaky! But it works. 

Here's my pattern configuration:

# add the user id to logging:logging:  pattern.console: >-
    %clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) 
    %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %8X{User} 
    %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n

I just had to add my MDC variable "User" to the default pattern.

03 January 2017

How do Microsoft's Azure Service Fabric and Pivotal's CloudFoundry compare?

How do Azure Service Fabric (ASF) and CloudFoundry (CF) compare? This is by no means neither an authoritative nor a complete comparison of the features, just a couple of quick observations. I base my statements on the resources mentioned at the end of my blog post.

Well, first of all, and obviously enough, ASF runs on Azure, nowhere else. CF runs on almost all Cloud providers out there, including Azure. Sounds like a great plus for CF when it comes to freedom of Cloud provider choice. The topic is vendor lock-in, and I have great respect for Microsoft's decision to provide CF as an alternative in Azure. I was not expecting this move from Microsoft, as their "normal" reaction in the past was to create something different to lock their customers in. It seems that they have changed their strategy completely. Now they simply compete with better services, and they seem to be quite sure about themselves.

On the CF side: We should not ignore that CF is not equal CF. The portal experience and the provided services can be quite different from one Cloud provider to the other. What remains is a application deployment model and core features that are standardized on CF. Moreover, authentication and authorization is not standardized on CF, and this creates a Cloud lock-in nevertheless. The only abstraction layer known to me for these aspects of an application is the Spring econiche with Spring Boot, Spring Security, and Spring Cloud. CF and Spring are a strong team, and the power behind its development is great.

ASF has some very strong selling points, too, though. It is used as the platform for many Microsoft internal platforms themselves, hence it is battle tested. Its "Reliable Actors" offering provides a very compelling distributed computing model. The "Reliable Actors" model does not provide the full Erlang/Akka Actor System model, but it provides the same abstraction layers. I believe that it most closely compares to AWS BeansTalk or AWS Lambda.

One more strong point for ASF is that developers can have their own local ASF instance by simply downloading the Service Fabric Developer SDK. You can install a local CF instance but it takes more than just running a setup-sdk.exe.

On the flip side of the coin: ASF creates technology and vendor lock-in. It currently supports only C#, although Java support is in the making. But Java will always be second-class citizen on ASF, new features will only be introduced after its introduction for .Net/C#.

Conclusions: CF is the "de facto standard" for PaaS. Hence a Public Cloud strategy for Enterprises could be to standardize on CloudFoundry in order to simplify the management of Public, Private and Internal Clouds. Other Public Cloud platforms should not be ruled out, but product development teams would then need to have very good arguments to move to a non-standard Cloud PaaS offering.

Resources:
What are your thoughts? Are you having a Multi Cloud strategy in your company? What is it? What are your experiences so far?

02 January 2017

Impact of Intego virus scanning on Spring Boot based development work on a Mac

Recently I was looking into slow startups of Spring Boot based applications on my MacBook Pro. Dave Syer created a performance testing suite at https://github.com/dsyer/spring-boot-startup-bench which I then used to gather comparable figures. As a result, I identified my Intego Internet Security X9 for Anti-Virus to be the source of the problem. The numbers were devastating - the test suite takes about 15 minutes without Anti-Virus, and about 49 minutes with Anti-Virus. That's about 3 times slower! Crazy.

I then took another spin with Bitdefender - and the numbers were much better - the test suite took about the same amount of time as without Anti-Virus software activated.

So if you come across performance problems while developing and testing your Spring Boot based applications and you happen to use Intego security software, that's the root cause.

Have you made similar experience with your OS and virus scanning software?

Should Software Architects write code?

Gregor answers this nicely in https://youtu.be/31qcPwAv8Zw . Yes, they should. But not to create production code, but to grasp the idea and ...