Netflix' data warehouse is based on Hadoop

I'm listening to Rüdiger from Microsoft. He explains what can be provided by Azure, HDInsight, a Big Data platform based on Hadoop.

This brings me to a blog entry I wanted to write for a while.

There are a couple of interesting articles on the Netflix blog about their use of Hadoop for their data scientists for analysis, and as a data warehouse platform:

• The Netflix Tech Blog: Genie 2.0: Second Wish Granted!
• The Netflix Tech Blog: Using Presto in our Big Data Platform on AWS
• The Netflix Tech Blog: Hadoop Platform as a Service in the Cloud

As you can see, they don't use relational databases as a backend for their data warehouse.

Would a Hadoop platform be a simple, cost effective data warehouse option?

The fascinating world of the Spring Boot logging pattern

I was recently adding Spring Cloud Sleuth to my Spring Boot applications, and I was wondering how I can make sure that my logging pattern is used but I will also have the sleuth elements in there. I was surprised that this seemed to work like magic, out of the box, nothing big to change for me, except to pull the default logigng pattern from the defaults.xml file, put it into my application.yml file in a nice way, done.

But as a curious person, I asked myself how this was possible. I found the explanation on https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html, under chapter 26.5. Heureka - Spring Cloud Sleuth uses the property "logging.pattern.level" to inject its own elements inot the logging pattern by replacing the level item. Sneaky! But it works. 

Here's my pattern configuration:

# add the user id to logging:logging:  pattern.console: >-
    %clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) 

    %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %8X{User} 

    %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n

I just had to add my MDC variable "User" to the default pattern.