bigdata

Big Data

Accelerating NiFi flows delivery: Part 1

While working in different contexts with NiFi, we have faced recurring challenges of development, maintenance and deployment optimization of NiFi flows. Whereas the basic approach suggests to manually duplicate pipelines for similar patterns, we believe that an automated approach is relevant for production purpose when it comes to implementing a significant amount of ingestion flows relying on a limited set of patterns or, more simply, when it comes to deploying these flows on different environments of execution. The ability to reach the right level of…

Read more
Archi & Techno

Industrial document classification with Deep Learning

Knowledge is a goldmine for companies. It comes in different shapes and forms: mainly documents (presentation slides and documentation) that allow businesses to share information with their customers and staff. The way companies harness this knowledge is central to their ability to develop their business successfully. One of the common ways to ease the access to this document base is to use search engines based on textual data. At OCTO, we have decided to use optical character recognition (OCR) solutions to extract this data, since…

Read more
Archi & Techno

Hadoop in my IT department: benchmark your cluster

The stress test is a very important step when you go live. Good stress tests help us to: ensure that the software meets its performances requirements ensure that the service will deliver a fast response time even under a heavy load get to now the scalability limits which in turn is useful to plan the next steps of the development Hadoop is not a web application, a database or a webservice. You don't stress test a Hadoop job with a heavy load. Instead, you need…

Read more
Archi & Techno

Hadoop in my IT department: How to plan a cluster?

Ok, you have decided to setup a Hadoop cluster for your business. Next step now, planning the cluster… But Hadoop is a complex stack and you might have many questions: HDFS deals with replication and Map Reduce create files… How can I plan my storage needs? How to plan my CPU needs? How to plan my memory needs? Should I consider different needs on some nodes of the cluster? I heard that Map Reduce moves its job code where the data to process is located……

Read more
Big Data

Introduction to Datastax Brisk : an Hadoop and Cassandra distribution

As the Apache Hadoop ecosystem grows while its core matures, there are now several companies providing business-class Hadoop distribution and services. While EMC, after it acquires Greenplum, seem the biggest player other companies such as Cloudera or MapR are also competing. This article introduces Datastax Brisk, an innovative Hadoop distribution that leverage Apache Hive data warehouse infrastructure on top of an HDFS-compatible storage layer, based on Cassandra. Brisk try to reconcile real-time applications with low-latency requirement (OLTP) and big data analytics (OLAP) in one system.…

Read more