DEV Community

Ali MSELMI
Ali MSELMI

Posted on

Structured logging with Serilog and Seq and ElasticSearch under Docker

#aspnetcore #docker #serilog #elasticsearch

This blog post demonstrates Structured Logging with Serilog, Seq, ElasticSearch and kibana under Docker containers.
This post is a follow up on the beginner post I wrote on Serilog.

If you are a newbie on Serilog then I will recommend that you give a quick read to official Serilog page and this blog post.

Example application

The complete below example shows serilog structured logging in a containerized web application with microservices style using docker, with events sent to the Seq and elastic search as well as a date-stamped rolling log file with the use of available below sinks:

  • Serilog.Sinks.File
  • Serilog.Sinks.Http
  • Serilog.Sinks.Seq
  • Serilog.Sinks.ElasticSearch

Create a new empty solution

To start, we need create a new solution using Visual studio or your favorite IDE.
Create a new empty solution

Create a new web Application project

Create new Web application project like below:
Create a new web Application project

Install the core Serilog package and the File, Seq and ElasticSearch sinks

In Visual Studio, open the Package Manager Console and type:

Install-Package Autofac
Install-Package Autofac.Extensions.DependencyInjection
Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson
Install-Package Microsoft.Extensions.DependencyInjection
Install-Package Microsoft.Extensions.Hosting
Install-Package Microsoft.Extensions.Logging
Install-Package Microsoft.Extensions.Options
Install-Package Newtonsoft.Json
Install-Package Serilog
Install-Package Serilog.AspNetCore
Install-Package Serilog.Extensions.Hosting
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Settings.AppSettings
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.ElasticSearch
Install-Package Serilog.Sinks.File
Install-Package Serilog.Sinks.Http
Install-Package Serilog.Sinks.Seq

Add the following code to Program.cs

Create GetConfigurationmethod 

 private static IConfiguration GetConfiguration() { var builder = new ConfigurationBuilder() .SetBasePath(Directory.GetCurrentDirectory()) .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true) .AddJsonFile($"appsettings.Development.json", optional: true) .AddEnvironmentVariables(); return builder.Build(); }
Enter fullscreen mode Exit fullscreen mode

Create CreateSerilogLoggermethod 

 private static ILogger CreateSerilogLogger(IConfiguration configuration) { var seqServerUrl = configuration["Serilog:SeqServerUrl"]; return new LoggerConfiguration() .MinimumLevel.Verbose() .Enrich.WithProperty("ApplicationContext", AppName) .Enrich.FromLogContext() .WriteTo.File("catalog.api.log.txt", rollingInterval: RollingInterval.Day) .WriteTo.Elasticsearch().WriteTo.Elasticsearch(ConfigureElasticSink(configuration, "Development")) .WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl) .CreateLogger(); }
Enter fullscreen mode Exit fullscreen mode

Create ConfigureElasticSinkmethod 

 private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration, string environment) { return new ElasticsearchSinkOptions(new Uri(configuration["Serilog:ElasticConfiguration"])) { BufferCleanPayload = (failingEvent, statuscode, exception) => { dynamic e = JObject.Parse(failingEvent); return JsonConvert.SerializeObject(new Dictionary<string, object>() { { "@timestamp",e["@timestamp"]}, { "level","Error"}, { "message","Error: "+e.message}, { "messageTemplate",e.messageTemplate}, { "failingStatusCode", statuscode}, { "failingException", exception} }); }, MinimumLogEventLevel = LogEventLevel.Verbose, AutoRegisterTemplate = true, AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7, CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true), IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}", EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | EmitEventFailureHandling.WriteToFailureSink | EmitEventFailureHandling.RaiseCallback }; }
Enter fullscreen mode Exit fullscreen mode

Create GetDefinedPorthelper method 

 private static int GetDefinedPort(IConfiguration config) { var port = config.GetValue("PORT", 80); return port; }
Enter fullscreen mode Exit fullscreen mode

Create CreateHostBuildermethod 

 private static IWebHost CreateHostBuilder(IConfiguration configuration, string[] args) => WebHost.CreateDefaultBuilder(args) .UseConfiguration(configuration) .CaptureStartupErrors(false) .ConfigureKestrel(options => { var httpPort = GetDefinedPort(configuration); options.Listen(IPAddress.Any, httpPort, listenOptions => { listenOptions.Protocols = HttpProtocols.Http1AndHttp2; }); }) .UseStartup<Startup>() .UseContentRoot(Directory.GetCurrentDirectory()) .UseSerilog() .Build();
Enter fullscreen mode Exit fullscreen mode

Create Main method 

 public static readonly string Namespace = typeof(Program).Namespace; public static readonly string AppName = Namespace.Substring(Namespace.LastIndexOf('.', Namespace.LastIndexOf('.') - 1) + 1); public static int Main(string[] args) { var configuration = GetConfiguration(); Log.Logger = CreateSerilogLogger(configuration); try { Log.Information("Configuring web host ({ApplicationContext})", AppName); var host = CreateHostBuilder(configuration, args); Log.Information("Starting web host ({ApplicationContext})", AppName); host.Run(); return 0; } catch (Exception ex) { Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName); return 1; } finally { Log.CloseAndFlush(); } }
Enter fullscreen mode Exit fullscreen mode

Add the following code to Startup.cs

Inject IConfigurationinto the constructure 

 public Startup(IConfiguration configuration) { Configuration = configuration; } public IConfiguration Configuration { get; }
Enter fullscreen mode Exit fullscreen mode

Create ConfigureServices method 

 public IServiceProvider ConfigureServices(IServiceCollection services) { services.AddCustomMVC(Configuration); var container = new ContainerBuilder(); container.Populate(services); return new AutofacServiceProvider(container.Build()); }
Enter fullscreen mode Exit fullscreen mode

Create Configuremethod 

 public void Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory) { var pathBase = Configuration["PATH_BASE"]; if (!string.IsNullOrEmpty(pathBase)) { loggerFactory.CreateLogger<Startup>().LogDebug("Using PATH BASE '{pathBase}'", pathBase); app.UsePathBase(pathBase); } app.UseCors("CorsPolicy"); app.UseRouting(); app.UseEndpoints(endpoints => { endpoints.MapDefaultControllerRoute(); endpoints.MapControllers(); }); }
Enter fullscreen mode Exit fullscreen mode

Create the CustomExtensionMethodsextension method 

 public static class CustomExtensionMethods { public static IServiceCollection AddCustomMVC(this IServiceCollection services, IConfiguration configuration) { services.AddControllers(options => { }).AddNewtonsoftJson(); services.AddCors(options => { options.AddPolicy("CorsPolicy", builder => builder .SetIsOriginAllowed((host) => true) .AllowAnyMethod() .AllowAnyHeader() .AllowCredentials()); }); return services; } }
Enter fullscreen mode Exit fullscreen mode

Add appsettings.json / appsettings.Development.json

It's very important to use the container name as ElasticSearchurl and not http://localhost:9200

 { "Serilog": { "SeqServerUrl": "http://seq", "LogstashgUrl": "http://locahost:8080", "ElasticConfiguration": "http://elasticsearch:9200", "MinimumLevel": { "Default": "Debug", "Override": { "Microsoft": "Debug", "CatalogAPI": "Debug", "MYSHOP": "Debug", "System": "Warning" } } } }
Enter fullscreen mode Exit fullscreen mode

Add Docker support to our solution

Add Docker compose project to our solution

Add Docker compose project to our solution

Add Dockerfile to the project 

 FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base WORKDIR /app EXPOSE 80 EXPOSE 443 FROM mcr.microsoft.com/dotnet/core/sdk:aspnet:3.1-buster AS build WORKDIR /src COPY "Services/Catalog/Catalog.API/CatalogAPI.csproj" "Services/Catalog/Catalog.API/CatalogAPI.csproj" COPY "docker-compose.dcproj" "docker-compose.dcproj" COPY "NuGet.config" "NuGet.config" RUN dotnet restore "MyShop.sln" COPY . . WORKDIR /src/Services/Catalog/Catalog.API RUN dotnet publish --no-restore -c Release -o /app FROM build AS publish FROM base AS final WORKDIR /app COPY --from=publish /app . ENTRYPOINT ["dotnet", "CatalogAPI.dll"]
Enter fullscreen mode Exit fullscreen mode

Add docker-compose.yml to the docker project 

 version: '3.4' services: seq: image: datalust/seq:latest catalog.api: image: ${REGISTRY:-myshop}/catalogapi build: context: . dockerfile: Services/Catalog/CatalogAPI/Dockerfile networks: elastic: driver: bridge volumes: elasticsearchdata: external: true
Enter fullscreen mode Exit fullscreen mode

Add Docker docker-compose.overrides.yml to the docker project 

 version: '3.4' services: seq: environment: - ACCEPT_EULA=Y ports: - "5340:80" elasticsearch: build: context: elk/elasticsearch/ volumes: - ./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro ports: - "9200:9200" - "9300:9300" environment: ES_JAVA_OPTS: "-Xmx256m -Xms256m" logstash: build: context: elk/logstash/ volumes: - ./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro - ./elk/logstash/pipeline:/usr/share/logstash/pipeline:ro ports: - "8080:8080" environment: LS_JAVA_OPTS: "-Xmx256m -Xms256m" depends_on: - elasticsearch kibana: build: context: elk/kibana/ volumes: - ./elk/kibana/config/:/usr/share/kibana/config:ro ports: - "5601:5601" depends_on: - elasticsearch catalog.api: environment: - ASPNETCORE_ENVIRONMENT=Development - ASPNETCORE_URLS=http://0.0.0.0:80 - PORT=80 - PATH_BASE=/catalog-api ports: - "4201:80"
Enter fullscreen mode Exit fullscreen mode

Create Configuration for ElasticSearchand Kibana

Create the folder structure

Create "elk" folder in the root folder of the solution with following structure
Create the folder structure

Create elasticsearch configuration

Under elk/elasticsearch create new Dockerfileand new config folder like bellow:
Create elasticsearch configuration
Add the following code to the Dockerfile

 # https://github.com/elastic/elasticsearch-docker FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2 # Add your elasticsearch plugins setup here # Example: RUN elasticsearch-plugin install analysis-icu
Enter fullscreen mode Exit fullscreen mode

Create new elasticsearch.yml under the config folder

 
 
   Default Elasticsearch configuration from elasticsearch-docker. 
 
   from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml 
 
#
 cluster.name: "docker-cluster"
 network.host: 0.0.0.0
 
# minimum_master_nodes need to be explicitly set when bound on a public IP
 # set to 1 to allow single node clusters
 # Details: https://github.com/elastic/elasticsearch/pull/17288
 discovery.zen.minimum_master_nodes: 1
 
## Use single node discovery in order to disable production mode and avoid bootstrap checks
 ## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
 #
 discovery.type: single-node
Enter fullscreen mode Exit fullscreen mode




Create kibana configuration

Under elk/kibana create new Dockerfileand new config folder like bellow:

Add the following code to the Dockerfile

 
# https://github.com/elastic/kibana-docker
 FROM docker.elastic.co/kibana/kibana-oss:7.6.2
 
# Add your kibana plugins setup here
 # Example: RUN kibana-plugin install <name|url>
 Create new kibana.yml under the config folder
 
## Default Kibana configuration from kibana-docker.
 ## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml
 #
 server.name: kibana
 server.host: "0"
 elasticsearch.hosts: "http://elasticsearch:9200"
Enter fullscreen mode Exit fullscreen mode




Conclusion

We have seen how to configure .net core web application to log in Elasticsearch, Kibanaand Sequsing Serilog. This solution offers many advantages and I invite you to discover the range of possibilities on the Elastic and Seq websites.
The sources are here feel free to clone the solution and play with it.

Top comments (1)

Collapse
 
moslemhady profile image
Moslem •

In serilog configuration, there is a property called "LogstashgUrl". Is this a typo? shouldn't it be "LogstashUrl"? or does it mean something I dont know?!