Apache HOP - Hello World !

 

In this blog post, we will see, how we can do a very simple fancy application - Hello World !

Apache HOP - Up and running

 

In this blog post, let's see how we can download and run Apache HOP (2.9) in Windows 11 environment.

1. Navigate to Apache HOP website. - https://hop.apache.org/ 

2. Click Download menu option and will navigate to Download page. -https://hop.apache.org/download/ 

3. Click  "apache-hop-client-2.9.0.zip" tod download the Apache HOP client application.

4. Extract the ZIP file

5. Click on hop-gui.bat file

6. You can see the following screen while starting Apache HOP.

Volia, here the welcome screen from Apache HOP GUI editor.

Happy Hopping !!!

Apache HOP - Core keywords

Here are the core keywords used in Apache HOP.

1. Pipelines : A pipelines is set of actions for the transform. It can read the data from the source, process and write the data.

2. HOP: To connect two actions, the HOP will be created. 

3. Worflow: It has starting and end points. Majorly consists set of pipelines to execute. 

4. Connectors: Connectors are bridges to connect external systems like database, files systems with Apache HOP. 

5. Plugins: Plug-ins are prebuilt tools to expand the Apache HOP's capabilities. 

Happy Hopping !

Apache HOP - Quick Introduction

What is Apache HOP ?

In simple, Apache HOP is a data engineering and orchestration platform. HOP is abbreviated as Hop Orchestration Platform. 

Apache HOP allows users to visually create data pipelines and workflows.

Why we need Apache HOP ?

Apache HOP helps users to automate data extraction from different data sources, performs data cleaning and data transformations and load them into other data sources.

Apache HOP vs Apache Airflow

Feature	Apache Hop	Apache Airflow
Focus	Data Integration & Orchestration	Workflow Orchestration & Scheduling
Strengths	- User-friendly visual interface - Pre-built transformations - Integrates with various data sources - Real-time data processing	- Flexible scheduling & dependency management - Supports diverse platforms (local, cloud) - Integrates with various data processing tools - Strong community & plugin ecosystem
Weaknesses	- Limited complex workflow scheduling	- Steeper learning curve (code-centric) - Requires more technical expertise
Platform	Windows, MacOS and Linux	MacOS and Linux
Language	Built on Java	Built on Python

Apache HOP vs Apache Nifi

Feature	Apache Hop	Apache NiFi
Focus	Data Integration & Orchestration	Data Ingestion & Stream Processing
Strengths	- User-friendly visual interface for building data pipelines - Pre-built transformations for data manipulation - Integrates with various data sources - Handles large data volumes (with powerful engines)	- Highly scalable for real-time data processing - Wide range of processors for data manipulation - Focuses on data flow & provenance - Distributed and fault-tolerant architecture
Weaknesses	- Less emphasis on streaming data compared to NiFi - Limited built-in scheduling capabilities (requires Airflow)	- Steeper learning curve for complex configurations - Requires more technical expertise for managing data flow
Platform	Windows, MacOS and Linux	Windows, MacOS and Linux
Language	Built on Java	Built on Java

Apache HOP vs Microsoft SSIS

Feature	Apache Hop	Microsoft SSIS
Type	Open-source data integration and orchestration platform	Proprietary data integration tool included with Microsoft SQL Server
Cost	Free and open-source	Paid (bundled with SQL Server licenses)
Deployment	On-premises or cloud (with cloud providers offering Hop environments)	On-premises only (requires a Windows Server)
User Interface	Visual interface with drag-and-drop functionality	Visual interface with a steeper learning curve
Data Sources / Destinations	Integrates with a wide variety of data sources and destinations	Primarily designed for integration with Microsoft products and databases
Real-time Processing	Supports real-time data processing with proper configuration	Primarily focused on batch data processing (ETL)
Scalability	Scales horizontally by adding more nodes	Scales vertically by adding more resources to a single server
Community & Support	Large and active open-source community with extensive online resources	Vendor support available through Microsoft licensing agreements

Apache HOP vs Azure Data Factory (ADF)

Feature	Apache Hop	Azure Data Factory (ADF)
Type	Open-source data integration and orchestration platform	Cloud-based, managed service from Microsoft Azure
Cost	Free and open-source	Paid service with various pricing tiers based on usage
Deployment	On-premises or cloud (with cloud providers offering Hop environments)	Cloud-based only (runs on Microsoft Azure)
User Interface	Visual interface with drag-and-drop functionality	Web-based visual interface with some code editing options
Data Sources / Destinations	Integrates with a wide variety of data sources and destinations	Primarily designed for integration with Azure services and other Microsoft products, but also supports various cloud and on-premises data sources
Real-time Processing	Supports real-time data processing with proper configuration	Supports real-time and batch data processing
Scalability	Scales horizontally by adding more nodes	Managed service that scales automatically based on your needs
Community & Support	Large and active open-source community with extensive online resources	Vendor support available through Microsoft Azure support channels

Endpoint Execution filters in Minimal API (.NET 7)

Endpoint Execution filters are new Minimal API in .NET 7 and this feature allows developers to perform validations before the actual API request is executed. The developers can validate the input parameters (from the body/query string or URL template) and validate user authentication information as well. 

These validations will make the system more stable and secure as the input parameters are validated before executing the actual code. 

There are three types of Endpoint Execution filters are available.

• Before Endpoint Execution Filter
• After Endpoint Execution Filter
• Short Circuit Execution Filter

To implement Endpoint execution filters, you need to use the IEndpointFilter interface and implement  

InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next ) with your custom logic code. You can access HTTPContext information by using  EndpointFilterInvocationContext context variable.

Before Endpoint Execution Filter:

As the name suggests, first the input parameters or the user authentication information are validated before executing the actual code. 

After Endpoint Execution Filter:

In this type of filter, the actual code is executed first and then the result of the execution will be used for further processing or transformation. 

Short-Circuit Execution Filter:

In this type of filter, the actual code will not be executed instead, the different logic will be performed and the response sent to the user.

The sample project can be downloaded from Github 

Happy Coding !!

Getting started with Minimal API - Create your project with dotnet CLI

The below blog post helps you to create a new Minimal API project using Visual Studio 2022 in a step-by-step manner. 

https://www.codingfreaks.net/2023/01/getting-started-with-minimal-api-first.html

You can use very simple dotnet CLI command to create a plain Minimal API project.

Syntax:

                dotnet new web

C:\Users\Murali\Documents\temp>dotnet new web -n HelloWorld

The above command creates a new Minimal API project with the name HelloWorld.

When you open the project in Visual Studio and it appears like the below.

Happy Coding !!

Getting started with Minimal API - The first project

This blog post explains, how to create your first Minimal API project using Visual Studio 2022 with .NET 7.

Step 1: Launch VS 2022 or higher 

Step 2: Choose Asp.net Core Web API

Step 3: Click Next and Enter the Project Name, Location name and Solution Name

 

Step 4: Choose Framework Version as .NET 7 & the minimum requirement is .NET 6.

Authentication Type: None

Configure for HTTPS: Yes. Do check the checkbox.

Enable Docker: No. Don’t Check the checkbox

Below are the important to enable Minimal API.

Use Controllers (uncheck to use Minimal API): No (Don’t check the checkbox)

Enable OpenAPI Support: Yes. Do check the checkbox.

Do not use top-level statements: Yes. Do check the checkbox.

 

Step 5: Once the application is successfully created, it appears like the below.

 

 

Step 6: Press F5, to run the application.

The Swagger UI page has been launched and you can view the Weatherforecast API as below.

Step 7: Click on the /weatherforecast API name, click Try it out, execute and you can see the results as below.

 

Happy Coding !!

New features of Minimal APIs in .NET 7

 Here are new features that are released on .NET 7 for Minimal API.

Create a new row in empty collection in Power Apps

 

Here is a syntax for creating a new row in the existing, empty collection.

Patch (<CollectionName>, defaults(<CollectionName>), {})

Here is an example of creating a new row in the existing, empty collection.

Patch (Users, defaults(Users), {})

Now, the Users collection will have one empty row and this updated User collection can be used to bind Gallery or datatable in Power Apps. 

Happy Coding !

Serverless components in Azure

Azure Serverless components are categorized as below.

Compute

                Azure Functions
                Azure App Services
                Azure Kubernetes Services (AKS)

Workflows and Integration
                Azure LogicApps
                Azure API Management
                Azure Event Grid
                Azure Service Bus

DevOps
                Azure DevOps

AI and ML
                Azure Cognitive Services
                Azure Bot Services
                Azure Machine Learning

Database
                Azure Cosmos DB
                Azure SQL Server Serverless

Storage
                Azure Blog Storage

Monitoring
                Azure Monitor

Analytics
                Azure Stream Analytics