Elasticsearch

Setup in Development Mode

Create Network & Start Container

docker network create elastic-network
docker run -d --name elasticsearch --net elastic-network -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:9.3.1

Generate Credentials

Run the following inside the container to generate passwords for default users:

docker exec -it elasticsearch bash
bin/elasticsearch-setup-passwords auto

...
Changed password for user elastic
PASSWORD elastic = <password>
  • Default User: elastic
  • Password: (Use the generated value)

Verify Connection

Visit https://localhost:9200. A successful JSON response with tagline "You Know, for Search" confirms it is running.

Kibana Setup

Start Container

docker run -d --name my-kibana --net elastic-network -p 5601:5601 kibana:9.3.1

Generate Enrollment Token

Required to link Kibana to Elasticsearch:

docker container exec -it elasticsearch bash
elasticsearch-create-enrollment-token -s kibana

eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTcyLjE4LjAuMjo5MjAwIl0sImZnciI6IjNjY2NkNTJlZjE1NjJhMzdkYzgwYTU3ZjA3OGMxMDUwNTk0YTZiZWRkYzcxOTY3NDBmOWQ0OWFmYzYwODU0YTQiLCJrZXkiOiJiN0tqLXB3Qk5FbXdfaGRmY2QyVTphQ1N5cTZUR0xzY2pQaXJrX01mWVV3In0=

Configuration & Login

  • Visit http://localhost:5601/
  • Paste in enroll token eyJ2ZXIiOiI4LjE0LjA...
  • Verification: Run the following to get your code: docker container exec -it kibana bin/kibana-verification-code
  • Log in using the elastic username and password generated in the previous step.

Ingest Pipeline for daily CSV files

Create pipeline

  • Go to Kibana Dev Tools (http://localhost:5601/app/dev_tools#/console) and run the following API:
PUT _ingest/pipeline/csv_parser
{
  "description": "Extracts custom timestamp from filename 2026-03-16_00-04-13",
  "on_failure": [
    {
      "set": {
        "field": "ingest_error",
        "value": ""
      }
    }
  ],
  "processors": [
    {
      "grok": {
        "field": "log.file.path",
        "patterns": ["%{GREEDYDATA}/%{YEAR:y}-%{MONTHNUM:m}-%{MONTHDAY:d}_%{HOUR:h}-%{MINUTE:min}-%{SECOND:s}\\.csv"]
      }
    },
    {
      "set": {
        "field": "_temp_ts",
        "value": "}-}-} }:content<!doctype html> <html lang="en" dir="auto"> <head><meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=5, user-scalable=no"> <meta name="description" content="Setup Latex Docker Container Create Dockerfile Dockerfile FROM leplusorg/latex RUN apt-get update &amp;amp;&amp;amp; apt-get install -y fonts-noto-..."> <meta name="revised" content="0ae8b51f6702b13609706b3fbd1ea50377ab5566"> <meta name="author" content="tsengyushiang"> <meta name="generator" content="JV-conseil/jekyll-theme-read-the-docs vJV-conseil/jekyll-theme-read-the-docs"><meta name="theme-color" content="#2980b9"> <title>Setup Latex Docker Container · notes</title> <meta name="twitter:title" content="Setup Latex Docker Container · notes"> <meta name="twitter:description" content="Setup Latex Docker Container Create Dockerfile Dockerfile FROM leplusorg/latex RUN apt-get update &amp;amp;&amp;amp; apt-get install -y fonts-noto-..."> <meta name="twitter:card" content="summary"> <meta name="twitter:site" content="@JVconseil"> <meta name="twitter:url" content="https://tsengyushiang.github.io/notes/Latex/setup.html"> <meta name="twitter:creator" content="@JVconseil"> <meta property="og:title" content="Setup Latex Docker Container · notes"> <meta property="og:description" content="Setup Latex Docker Container Create Dockerfile Dockerfile FROM leplusorg/latex RUN apt-get update &amp;amp;&amp;amp; apt-get install -y fonts-noto-..."> <meta property="og:locale" content="en"> <meta property="og:url" content="https://tsengyushiang.github.io/notes/Latex/setup.html"> <meta property="og:type" content="article"> <meta property="og:image" content="https://tsengyushiang.github.io/notes/assets/images/social-media-preview.png"> <meta property="article:author" content="tsengyushiang"> <meta property="article:published_time" content="2020-01-03T16:34:43+00:00"> <meta property="article:modified_time" content="2026-04-07T02:28:09+00:00"> <script type="application/ld+json"> { "@context": "https://schema.org", "@type": "Article", "mainEntityOfPage": { "@type": "WebPage", "@id": "https://tsengyushiang.github.io/notes/Latex/setup.html" }, "headline": "Setup Latex Docker Container · notes", "image": [], "author": { "@type": "Person", "name": "tsengyushiang" }, "datePublished": "2020-01-03T16:34:43+00:00", "dateModified": "2026-04-07T02:28:09+00:00", "publisher": { "@type": "User", "name": "tsengyushiang", "logo": { "@type": "ImageObject", "url": "https://avatars.githubusercontent.com/u/27216619?v=4" } }, "description": "Setup Latex Docker Container Create Dockerfile Dockerfile FROM leplusorg/latex RUN apt-get update &amp;amp;&amp;amp; apt-get install -y fonts-noto-..." } </script> <link rel="dns-prefetch" href="https://cdn.jsdelivr.net"><link rel="canonical" href="https://tsengyushiang.github.io/notes/Latex/setup.html"><link rel="shortcut icon" type="image/x-icon" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/favicon.ico"><link rel="icon" type="image/svg+xml" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/favicon.svg"> <link rel="icon" type="image/png" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/favicon-16x16.png" sizes="16x16"> <link rel="icon" type="image/png" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/favicon-32x32.png" sizes="32x32"> <link rel="icon" type="image/png" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/favicon-96x96.png" sizes="96x96"><link rel="mask-icon" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/favicon.svg" color="#2980b9"><link rel="apple-touch-icon" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/images/apple-touch-icon-300x300.jpg"> <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/css/theme.min.css"><script> window.ui = { title: "notes", baseurl: "/notes", i18n: { search_results: "Search Results", search_results_found: "Search finished, found # page(s) matching the search query.", search_results_not_found: "Your search did not match any documents, please make sure that all characters are spelled correctly!" } }; </script> </head> <body class="container"><div class="sidebar-wrap overflow-hidden"> <div class="sidebar height-full overflow-y-scroll overflow-x-hidden"> <div class="header d-flex flex-column p-3 text-center"> <div class="title pb-1"> <a class="h4 no-underline py-1 px-2 rounded-1" href="/notes/" title=""> <i class="fa fa-home"></i> notes </a> </div> <span class="version"></span> <form class="search pt-2" action="/notes/search.html" method="get" autocomplete="off"> <input class="form-control input-block input-sm" type="text" name="q" placeholder="Search docs..."> </form> </div> <div class="toctree py-2" data-spy="affix" role="navigation" aria-label="main navigation"> <ul> </ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/Latex/"> LaTex </a><ul> <li class="toc level-1 current" data-sort="" data-level="1"> <a class="d-flex flex-items-baseline current" href="/notes/Latex/setup.html">Setup Latex Docker Container</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/backend/"> Backend </a><ul> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/backend/elasticsearch.html">Elasticsearch</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/backend/nodejs.html">Nodejs</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/backend/python.html">Python</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/backend/robotframework.html">Robot Framework</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/deploy/"> Deploy </a><ul> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/deploy/electron.html">Electron</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/deploy/gitops.html">GitOps</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/deploy/k3s.html">K3s</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/deploy/vercel.html">Vercel</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/docker/"> Docker </a><ul> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/docker/commands.html">Commands</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/docker/mailserver.html">Mailserver</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/docker/nginx.html">Nginx</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/docker/ngrok.html">Ngrok</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/docker/others.html">Others</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/docker/sonarqube.html">Sonarqube</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/frontend/"> Frontend </a><ul> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/code_generator.html">Code Generator</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/codingStyle.html">Coding Style</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/css.html">CSS</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/helpers.html">Helpers</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/jsdoc.html">Jsdoc</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/matomo.html">Matomo</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/meta.html">Meta</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/monorepo.html">Monorepo</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/next_i18next.html">next-i18next</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/nextjs.html">NextJs</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/package_manager.html">Package Manager</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/packages.html">Packages</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/react_bundle_widget.html">React Bundle Widget</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/react_components.html">React Components</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/react_typescript.html">Typescript</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/redux-toolkit.html">Redux Toolkit</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/reduxSaga.html">redux-saga</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/redux_context.html">Redux &amp; Context Hook</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/security.html">Security</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/storybook.html">Storybook</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/styledComponents.html">Styled-components</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/threejs.html">Three.js</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/unit-testing.html">Unit Testing</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/frontend/vanillaJs.html">Vanilla Js</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/git/"> Git </a><ul> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/commands.html">Commands</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/docker-container-package.html">Docker container package</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/github.html">GitHub</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/gitlab.html">GitLab</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/private-npm-package.html">Private NPM Package</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/revert-merge-request.html">Revert Merge Request</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/ssh.html">Github SSH</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/undo-force-push.html">Undo force push</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/git/workflow.html">Fork Workflow</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/google/"> Google </a><ul> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/google/google-appa-script.html">Google Apps Script</a> </li> <li class="toc level-1 " data-sort="" data-level="1"> <a class="d-flex flex-items-baseline " href="/notes/google/spread-sheet-api.html">Spread Sheet API</a> </li></ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/tools/"> Tools </a><ul> </ul> <a class="caption d-block text-uppercase no-wrap px-2 py-0" href="/notes/vscode/"> Vscode </a><ul> </ul> </div> </div> </div> <div class="content-wrap"> <div class="header d-flex flex-justify-between p-2 hide-lg hide-xl" aria-label="top navigation"> <button id="toggle" aria-label="Toggle menu" class="btn-octicon p-2 m-0 text-white" type="button"> <i class="fa fa-bars"></i> </button> <div class="title flex-1 d-flex flex-justify-center"> <a class="h4 no-underline py-1 px-2 rounded-1" href="/notes/">notes</a> </div> </div> <div class="content p-3 p-sm-5"> <div class="navigation-top d-flex flex-justify-between"> <ul class="breadcrumb" role="navigation" aria-label="breadcrumbs navigation"> <li class="breadcrumb-item"> <a class="no-underline" href="/notes/" title="/"> <i class="fa fa-home"></i> </a> </li><li class="breadcrumb-item" ><a href="/notes/Latex/">Latex</a></li><li class="breadcrumb-item" aria-current="page">setup.md</li></ul> <a class="edit" href="https://github.com/tsengyushiang/notes/edit/master/Latex/setup.md" title="Edit on GitHub" rel="noreferrer" target="_blank"> <i class="fa fa-edit"></i> </a> </div> <hr> <div role="main" itemscope="itemscope" itemtype="https://schema.org/Article"> <div class="markdown-body" itemprop="articleBody"> <h1 id="setup-latex-docker-container">Setup Latex Docker Container</h1> <h3 id="create-dockerfile">Create Dockerfile</h3> <p><code class="language-plaintext highlighter-rouge notranslate notranslate">Dockerfile</code></p> <div class="language-Dockerfile highlighter-rouge notranslate notranslate"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> leplusorg/latex</span>
<span class="k">RUN </span>apt-get update <span class="o">&amp;&amp;</span> apt-get <span class="nb">install</span> <span class="nt">-y</span> fonts-noto-cjk
</code></pre> </div></div> <ul> <li>Package <code class="language-plaintext highlighter-rouge notranslate notranslate">fonts-noto-cjk</code> is used to compile chinese, if you don't need just remove the command.</li> </ul> <h3 id="build-image">Build Image</h3> <div class="language-plaintext highlighter-rouge notranslate notranslate"><div class="highlight"><pre class="highlight"><code>docker build -t my-latex-image .
</code></pre> </div></div> <h3 id="preparing-latex-files">Preparing LaTex Files</h3> <ul> <li> <p>Copy example from following link and renamed it <code class="language-plaintext highlighter-rouge notranslate notranslate">entry.tex</code>.</p> <ul> <li><a href="/notes/Latex/examples/forest.html">Tree Graph</a></li> </ul> </li> </ul> <h3 id="compile-latex-to-pdf">Compile LaTex to PDF</h3> <div class="language-plaintext highlighter-rouge notranslate notranslate"><div class="highlight"><pre class="highlight"><code>docker run --rm -v ./:/tmp my-latex-image latexmk -outdir=/tmp -pdfxe /tmp/entry.tex
</code></pre> </div></div> <ul> <li><code class="language-plaintext highlighter-rouge notranslate notranslate">-pdfxe</code> use compiler XeLaTex, I use it for package <code class="language-plaintext highlighter-rouge notranslate notranslate">\usepackage{xeCJK}</code></li> <li>You can use <code class="language-plaintext highlighter-rouge notranslate notranslate">-pdf</code> instead to use compiler pdfLaTex.</li> </ul> </div> </div> <hr> <div class="copyright text-center text-gray" role="contentinfo"> <i class="fa fa-copyright"></i> <span class="time">2020-2026,</span> <a class="text-gray" href="https://github.com/tsengyushiang" rel="noreferrer" target="_blank">tsengyushiang</a> Revision <a class="text-gray" href="https://github.com/tsengyushiang/notes/commit/0ae8b51f6702b13609706b3fbd1ea50377ab5566" title="0ae8b51f6702b13609706b3fbd1ea50377ab5566" rel="noreferrer" target="_blank">0ae8b51</a> <br> <div class="generator"> Built with <a href="https://pages.github.com" rel="noreferrer" target="_blank" title="github-pages v232">GitHub Pages</a> using a <a href="https://github.com/JV-conseil/jekyll-theme-read-the-docs" rel="noreferrer" target="_blank" title="JV-conseil/jekyll-theme-read-the-docs vJV-conseil/jekyll-theme-read-the-docs">theme</a> provided by <a href="https://github.com/JV-conseil/" rel="noreferrer" target="_blank">JV conseil</a>. </div> <a href="https://github.com/sponsors/JV-conseil"><img style="vertical-align:middle" src="https://img.shields.io/static/v1?label=Sponsor&amp;message=%E2%9D%A4&amp;logo=GitHub&amp;color=%23fe8e86" alt="Become a sponsor to JV-conseil"></a> </div> </div> </div> <div class="addons-wrap d-flex flex-column overflow-y-auto"> <div class="status d-flex flex-justify-between p-2"> <div class="title p-1"> <i class="fa fa-book"></i> notes </div> <div class="branch p-1"> <span class="name"> master </span> <i class="fa fa-caret-down"></i> </div> </div> <div class="addons d-flex flex-column height-full p-2 d-none"> <dl> <dt>GitHub</dt> <dd> <a href="https://github.com/tsengyushiang/notes" title="Stars: 0"> <i class="fa fa-github"></i> Homepage </a> </dd> <dd> <a href="https://github.com/tsengyushiang/notes/issues" title="Open issues: 0"> <i class="fa fa-question-circle-o"></i> Issues </a> </dd> <dd> <a href="https://github.com/tsengyushiang/notes/zipball/master" title="Size: 54677 Kb"> <i class="fa fa-download"></i> Download </a> </dd> </dl> <hr> <div class="license f6 pb-2"> This <a href="/notes/" title="notes">Software</a> is under the terms of <a href="https://github.com/tsengyushiang/notes">The Unlicense</a>. </div> </div> </div> <!-- <script src="https://cdn.jsdelivr.net/npm/jquery@3/dist/jquery.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/dompurify@3/dist/purify.min.js"></script> --> <script src="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/lib/jquery.min.js"></script> <script src="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/lib/purify.min.js"></script><script src="https://cdn.jsdelivr.net/gh/JV-conseil/jekyll-theme-read-the-docs/assets/js/theme.min.js"></script> <!-- @prettify-ignore --><!-- @prettify-ignore --></body> </html>}:}"
      }
    },
    {
      "date": {
        "field": "_temp_ts",
        "formats": ["yyyy-MM-dd HH:mm:ss"],
        "target_field": "@timestamp"
      }
    },
    {
      "csv": {
        "field": "message",
        "target_fields": ["column1", "column2"],
        "separator": ","
      }
    },
    {
      "remove": {
        "field": ["message", "_temp_ts", "y", "m", "d", "h", "min", "s"],
        "ignore_missing": true
      }
    }
  ]
}
  • Verify the pipeline with the following simulation data. If the output displays a clean "timestamp": "2026-03-16T00:04:13.000Z", you're all set!
POST _ingest/pipeline/csv_parser/_simulate
{
  "docs": [
    {
      "_source": {
        "log": { "file": { "path": "/usr/share/filebeat/csv_data/2026-03-16_00-04-13.csv" } },
        "message": "Value1,Value2"
      }
    }
  ]
}
  • Index Template Setup

Configures modular mappings and automated data stream settings for all my-index* indices to ensure consistent schema and ingestion processing.

PUT _component_template/my-data-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "due_date": { "type": "date" },
        "status": { "type": "keyword" },
        "owner": { "type": "keyword" }
      }
    }
  }
}

PUT _index_template/my_index_template
{
  "index_patterns": ["my-index*"],
  "data_stream": { },
  "composed_of": ["my-data-mappings"], 
  "priority": 500,
  "template": {
    "settings": {
      "index.default_pipeline": "csv_parser"
    }
  }
}

Setup Filebeat

  • Configure filebeat.yml to monitor file changes:
filebeat.inputs:
  - type: filestream   
    id: daily-csv-pipeline
    enabled: true
    paths:
      - /usr/share/filebeat/csv_data/*.csv
    processors:
      - replace:
          fields:
            - field: "message"
              pattern: '"'
              replacement: ""

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "elastic"
  password: "<password>"
  ssl.verification_mode: "none"
  pipeline: "csv_parser"
  index: "my-index"

setup.template.enabled: false
  • Run the Filebeat container:
docker run -d \
  --name filebeat \
  --user root \
  --net elastic-network \
  -v "$(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \
  -v "$(pwd)/csv_data:/usr/share/filebeat/csv_data:ro" \
  -v "$(pwd)/filebeat_data:/usr/share/filebeat/data" \
  docker.elastic.co/beats/filebeat:9.3.1 \
  filebeat run -e --strict.perms=false
  • Add a CSV to /csv_data and verify it appears in Elasticsearch.

Debug Filebeat container issues:

# Test configuration validity
docker exec -it filebeat filebeat test config

# Test Elasticsearch connectivity
docker exec -it filebeat filebeat test output

# Monitor logs to confirm the harvester is active
docker logs filebeat | grep "harvester"