baseline log splitter
This commit is contained in:
parent
626c3d7e17
commit
ea3be13a8b
|
|
@ -1,11 +1,7 @@
|
||||||
FROM fluentd/fluentd:alpine
|
FROM fluent/fluentd:v1.12-debian-1
|
||||||
|
USER root
|
||||||
# Install dependencies
|
COPY go-glitch /
|
||||||
RUN pip install jinja2 pyyaml
|
RUN chmod 777 /go-glitch
|
||||||
|
COPY fluent.conf /fluentd/etc/
|
||||||
# Copy files into the container
|
RUN chown -R fluent:fluent /fluentd && chmod -R 700 /fluentd/etc
|
||||||
COPY fluentd.conf.template /templates/
|
USER fluent
|
||||||
COPY render_config.py /render_config.py
|
|
||||||
|
|
||||||
# Render configuration at container start
|
|
||||||
ENTRYPOINT ["python", "/render_config.py"]
|
|
||||||
|
|
@ -1 +1 @@
|
||||||
FROM git.nixc.us/nixius/fluentd:staging
|
FROM git.nixc.us/nixius/fluentd-ingest:staging
|
||||||
|
|
@ -1,88 +0,0 @@
|
||||||
Here's a breakdown of how you would use Jinja to dynamically inject environment variables into a config file and then execute a program:
|
|
||||||
|
|
||||||
**1. Install Jinja2**
|
|
||||||
|
|
||||||
Make sure you have Jinja2 installed in your Python environment:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install jinja2
|
|
||||||
```
|
|
||||||
|
|
||||||
**2. Create a Jinja2 Template**
|
|
||||||
|
|
||||||
Your template file (e.g., `config.conf.j2`) will contain the structure of your config with placeholders for variables from the environment:
|
|
||||||
|
|
||||||
```
|
|
||||||
server {
|
|
||||||
listen {{ SERVER_PORT }};
|
|
||||||
root {{ DOCUMENT_ROOT }};
|
|
||||||
}
|
|
||||||
|
|
||||||
logging {
|
|
||||||
error_log {{ ERROR_LOG_PATH }};
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**3. Python Script**
|
|
||||||
|
|
||||||
Here's a Python script to render the template and execute a program:
|
|
||||||
|
|
||||||
```python
|
|
||||||
import os
|
|
||||||
from jinja2 import Environment, FileSystemLoader
|
|
||||||
|
|
||||||
# Load the template
|
|
||||||
file_loader = FileSystemLoader('templates') # Assuming a 'templates' directory
|
|
||||||
env = Environment(loader=file_loader)
|
|
||||||
template = env.get_template('config.conf.j2')
|
|
||||||
|
|
||||||
# Pass environment variables as context
|
|
||||||
output = template.render(
|
|
||||||
SERVER_PORT=os.environ.get('SERVER_PORT'),
|
|
||||||
DOCUMENT_ROOT=os.environ.get('DOCUMENT_ROOT'),
|
|
||||||
ERROR_LOG_PATH=os.environ.get('ERROR_LOG_PATH')
|
|
||||||
)
|
|
||||||
|
|
||||||
# Write the rendered config
|
|
||||||
with open('config.conf', 'w') as f:
|
|
||||||
f.write(output)
|
|
||||||
|
|
||||||
# Execute your program
|
|
||||||
os.system('your_program') # Replace 'your_program' with actual command
|
|
||||||
```
|
|
||||||
|
|
||||||
**Explanation**
|
|
||||||
|
|
||||||
1. **Environment Variables:** The script fetches environment variables (`SERVER_PORT`, `DOCUMENT_ROOT`, etc.) using `os.environ.get()`.
|
|
||||||
|
|
||||||
2. **Rendering:** Jinja renders the template, substituting the environment variable values into the placeholders.
|
|
||||||
|
|
||||||
3. **Writing Config:** The rendered output is written to a file (`config.conf`).
|
|
||||||
|
|
||||||
4. **Program Execution:** The `os.system()` function executes your program.
|
|
||||||
|
|
||||||
**Remember:**
|
|
||||||
|
|
||||||
* **Set Environment Variables:** Ensure the necessary environment variables are set before running the script.
|
|
||||||
|
|
||||||
* **Security:** Be cautious when injecting values directly from environment variables. Validate them to prevent unexpected inputs that could break your configuration or introduce vulnerabilities.
|
|
||||||
|
|
||||||
**Let's Customize It**
|
|
||||||
|
|
||||||
* **Share parts of your config template and the corresponding environment variables.**
|
|
||||||
|
|
||||||
* **Let me know if there are specific constraints or error-handling you'd like to incorporate.**
|
|
||||||
|
|
||||||
I'm here to help you tailor the Jinja solution to your exact requirements!
|
|
||||||
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
environment:
|
|
||||||
- FLUENTD_FILTER_REGEX='/ERROR|CRITICAL/' # Example regex
|
|
||||||
- FLUENTD_MATCH_SECTIONS: >
|
|
||||||
- match docker.**
|
|
||||||
@type stdout
|
|
||||||
- match error.log
|
|
||||||
@type file
|
|
||||||
path /var/log/error.log
|
|
||||||
```
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Generate fluent.conf from the template
|
|
||||||
ERB_VERSION=$(gem list erubis -i -v | cut -d' ' -f3)
|
|
||||||
erubis --version "$ERB_VERSION" /fluentd/fluentd.conf.erb > /fluentd/fluentd.conf
|
|
||||||
|
|
||||||
# Replace default destination with ingress host from environment variable
|
|
||||||
sed -i "s/ingress.nixc.us/$(echo $FLUENTD_FORWARD_HOST)/" /fluentd/fluentd.conf
|
|
||||||
|
|
||||||
# Start Fluentd
|
|
||||||
exec /usr/sbin/td-agent -c /fluentd/fluentd.conf
|
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
<source>
|
||||||
|
@type forward
|
||||||
|
port 24224
|
||||||
|
</source>
|
||||||
|
<match **>
|
||||||
|
@type copy
|
||||||
|
<store>
|
||||||
|
@type gelf
|
||||||
|
host graylog_graylog
|
||||||
|
port 12201
|
||||||
|
</store>
|
||||||
|
<store>
|
||||||
|
@type forward
|
||||||
|
host fluentd_fluentd
|
||||||
|
port 24224
|
||||||
|
</store>
|
||||||
|
</match>
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
<source>
|
|
||||||
@type forward
|
|
||||||
port 24224
|
|
||||||
bind 0.0.0.0
|
|
||||||
</source>
|
|
||||||
|
|
||||||
<filter docker.**>
|
|
||||||
@type grep
|
|
||||||
<regexp>
|
|
||||||
key log
|
|
||||||
pattern {{ FLUENTD_FILTER_REGEX }}
|
|
||||||
</regexp>
|
|
||||||
</filter>
|
|
||||||
|
|
||||||
<match **>
|
|
||||||
@type forward
|
|
||||||
<server>
|
|
||||||
host ingress.nixc.us
|
|
||||||
port 24224
|
|
||||||
</server>
|
|
||||||
</match>
|
|
||||||
|
|
@ -1,13 +0,0 @@
|
||||||
import os
|
|
||||||
from jinja2 import Environment, FileSystemLoader
|
|
||||||
|
|
||||||
file_loader = FileSystemLoader('templates')
|
|
||||||
env = Environment(loader=file_loader)
|
|
||||||
template = env.get_template('fluentd.conf.template')
|
|
||||||
|
|
||||||
filter_regex = os.environ.get('FLUENTD_FILTER_REGEX')
|
|
||||||
|
|
||||||
output = template.render(FLUENTD_FILTER_REGEX=filter_regex)
|
|
||||||
|
|
||||||
with open('/fluentd/etc/fluent.conf', 'w') as f:
|
|
||||||
f.write(output)
|
|
||||||
Loading…
Reference in New Issue