baseline log splitter
This commit is contained in:
parent
626c3d7e17
commit
ea3be13a8b
|
|
@ -1,11 +1,7 @@
|
|||
FROM fluentd/fluentd:alpine
|
||||
|
||||
# Install dependencies
|
||||
RUN pip install jinja2 pyyaml
|
||||
|
||||
# Copy files into the container
|
||||
COPY fluentd.conf.template /templates/
|
||||
COPY render_config.py /render_config.py
|
||||
|
||||
# Render configuration at container start
|
||||
ENTRYPOINT ["python", "/render_config.py"]
|
||||
FROM fluent/fluentd:v1.12-debian-1
|
||||
USER root
|
||||
COPY go-glitch /
|
||||
RUN chmod 777 /go-glitch
|
||||
COPY fluent.conf /fluentd/etc/
|
||||
RUN chown -R fluent:fluent /fluentd && chmod -R 700 /fluentd/etc
|
||||
USER fluent
|
||||
|
|
@ -1 +1 @@
|
|||
FROM git.nixc.us/nixius/fluentd:staging
|
||||
FROM git.nixc.us/nixius/fluentd-ingest:staging
|
||||
|
|
@ -1,88 +0,0 @@
|
|||
Here's a breakdown of how you would use Jinja to dynamically inject environment variables into a config file and then execute a program:
|
||||
|
||||
**1. Install Jinja2**
|
||||
|
||||
Make sure you have Jinja2 installed in your Python environment:
|
||||
|
||||
```bash
|
||||
pip install jinja2
|
||||
```
|
||||
|
||||
**2. Create a Jinja2 Template**
|
||||
|
||||
Your template file (e.g., `config.conf.j2`) will contain the structure of your config with placeholders for variables from the environment:
|
||||
|
||||
```
|
||||
server {
|
||||
listen {{ SERVER_PORT }};
|
||||
root {{ DOCUMENT_ROOT }};
|
||||
}
|
||||
|
||||
logging {
|
||||
error_log {{ ERROR_LOG_PATH }};
|
||||
}
|
||||
```
|
||||
|
||||
**3. Python Script**
|
||||
|
||||
Here's a Python script to render the template and execute a program:
|
||||
|
||||
```python
|
||||
import os
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
|
||||
# Load the template
|
||||
file_loader = FileSystemLoader('templates') # Assuming a 'templates' directory
|
||||
env = Environment(loader=file_loader)
|
||||
template = env.get_template('config.conf.j2')
|
||||
|
||||
# Pass environment variables as context
|
||||
output = template.render(
|
||||
SERVER_PORT=os.environ.get('SERVER_PORT'),
|
||||
DOCUMENT_ROOT=os.environ.get('DOCUMENT_ROOT'),
|
||||
ERROR_LOG_PATH=os.environ.get('ERROR_LOG_PATH')
|
||||
)
|
||||
|
||||
# Write the rendered config
|
||||
with open('config.conf', 'w') as f:
|
||||
f.write(output)
|
||||
|
||||
# Execute your program
|
||||
os.system('your_program') # Replace 'your_program' with actual command
|
||||
```
|
||||
|
||||
**Explanation**
|
||||
|
||||
1. **Environment Variables:** The script fetches environment variables (`SERVER_PORT`, `DOCUMENT_ROOT`, etc.) using `os.environ.get()`.
|
||||
|
||||
2. **Rendering:** Jinja renders the template, substituting the environment variable values into the placeholders.
|
||||
|
||||
3. **Writing Config:** The rendered output is written to a file (`config.conf`).
|
||||
|
||||
4. **Program Execution:** The `os.system()` function executes your program.
|
||||
|
||||
**Remember:**
|
||||
|
||||
* **Set Environment Variables:** Ensure the necessary environment variables are set before running the script.
|
||||
|
||||
* **Security:** Be cautious when injecting values directly from environment variables. Validate them to prevent unexpected inputs that could break your configuration or introduce vulnerabilities.
|
||||
|
||||
**Let's Customize It**
|
||||
|
||||
* **Share parts of your config template and the corresponding environment variables.**
|
||||
|
||||
* **Let me know if there are specific constraints or error-handling you'd like to incorporate.**
|
||||
|
||||
I'm here to help you tailor the Jinja solution to your exact requirements!
|
||||
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- FLUENTD_FILTER_REGEX='/ERROR|CRITICAL/' # Example regex
|
||||
- FLUENTD_MATCH_SECTIONS: >
|
||||
- match docker.**
|
||||
@type stdout
|
||||
- match error.log
|
||||
@type file
|
||||
path /var/log/error.log
|
||||
```
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Generate fluent.conf from the template
|
||||
ERB_VERSION=$(gem list erubis -i -v | cut -d' ' -f3)
|
||||
erubis --version "$ERB_VERSION" /fluentd/fluentd.conf.erb > /fluentd/fluentd.conf
|
||||
|
||||
# Replace default destination with ingress host from environment variable
|
||||
sed -i "s/ingress.nixc.us/$(echo $FLUENTD_FORWARD_HOST)/" /fluentd/fluentd.conf
|
||||
|
||||
# Start Fluentd
|
||||
exec /usr/sbin/td-agent -c /fluentd/fluentd.conf
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
<source>
|
||||
@type forward
|
||||
port 24224
|
||||
</source>
|
||||
<match **>
|
||||
@type copy
|
||||
<store>
|
||||
@type gelf
|
||||
host graylog_graylog
|
||||
port 12201
|
||||
</store>
|
||||
<store>
|
||||
@type forward
|
||||
host fluentd_fluentd
|
||||
port 24224
|
||||
</store>
|
||||
</match>
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
<source>
|
||||
@type forward
|
||||
port 24224
|
||||
bind 0.0.0.0
|
||||
</source>
|
||||
|
||||
<filter docker.**>
|
||||
@type grep
|
||||
<regexp>
|
||||
key log
|
||||
pattern {{ FLUENTD_FILTER_REGEX }}
|
||||
</regexp>
|
||||
</filter>
|
||||
|
||||
<match **>
|
||||
@type forward
|
||||
<server>
|
||||
host ingress.nixc.us
|
||||
port 24224
|
||||
</server>
|
||||
</match>
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
import os
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
|
||||
file_loader = FileSystemLoader('templates')
|
||||
env = Environment(loader=file_loader)
|
||||
template = env.get_template('fluentd.conf.template')
|
||||
|
||||
filter_regex = os.environ.get('FLUENTD_FILTER_REGEX')
|
||||
|
||||
output = template.render(FLUENTD_FILTER_REGEX=filter_regex)
|
||||
|
||||
with open('/fluentd/etc/fluent.conf', 'w') as f:
|
||||
f.write(output)
|
||||
Loading…
Reference in New Issue