- Notifications
You must be signed in to change notification settings - Fork 311
Description
Handling feed detection
During the init method the parser try to determine the feed type detection type [0]:
intelmq/intelmq/bots/parsers/shadowserver/parser.py
Lines 42 to 43 in 7674949
| def init(self): | |
| if self.feedname is not None: |
If the self.feedname isn't set, then the mode is detect [1]:
intelmq/intelmq/bots/parsers/shadowserver/parser.py
Lines 53 to 54 in 7674949
| else: | |
| self._mode = 'detect' |
Later, during processing a report, the feed name is detected and eventually set [2]:
intelmq/intelmq/bots/parsers/shadowserver/parser.py
Lines 76 to 79 in 7674949
| if not retval: | |
| raise ValueError('Could not get a config for {!r}, check the documentation.' | |
| ''.format(self.report_name)) | |
| self.feedname, self._sparser_config = retval |
Reloading bot
Then, during the reload (not restart) happens, the bot is re-initialized without creating a new object:
Lines 271 to 285 in 7674949
| def __handle_sighup(self): | |
| """ | |
| Handle SIGHUP. | |
| """ | |
| if not self.__sighup.is_set(): | |
| return False | |
| self.logger.info('Handling SIGHUP, initializing again now.') | |
| self.__disconnect_pipelines() | |
| try: | |
| self.shutdown() # disconnects, stops threads etc | |
| except Exception: | |
| self.logger.exception('Error during shutdown of bot.') | |
| self.logger.handlers = [] # remove all existing handlers | |
| self.__sighup.clear() | |
| self.__init__(self.__bot_id_full, sighup_event=self.__sighup) |
This mechanism is used during log rotation by our standard Debian configuration:
intelmq/contrib/logrotate/intelmq
Line 12 in 7674949
| sudo -u intelmq /usr/local/bin/intelmqctl --quiet reload |
and of course re-load the configuration, refreshing all parameters set in the config file [3]:
Lines 786 to 788 in 7674949
| for option, value in params.get('parameters', {}).items(): | |
| setattr(self, option, value) | |
| self.__log_configuration_parameter("runtime", option, value) |
Failing edge case
Requirements
- Assuming you want parser to detect the feed, so you didn't set any value in
parametrs.feednamein the parser config. - You have also installed our Debian package or manually configured log rotation with reloading.
- Your IntelMQ instance is working long enough for log rotation to occur.
Case steps
- After starting bot, the mode is correctly detect [1], and then handling every report a correct feed is detected and set [2].
- The log rotation occurs, and the parser is reloaded.
- The configuration is refreshed, overriding all fields set in the config file [3].
- Bot
initmethod is called once more and tries to detect mode, butself.feednameis still set with the last value before reload, and thefixedmode is detected [0]. - On every next report handled, the last previously detected feed is used
Workarounds
Set the feedname as null in the config. However, due to the IntelMQ Manager issue with nulls certtools/intelmq-manager#294 this may be changed to an empty string and case the fail again.
Fixes
- The quickest - clean the
feednamein bot'sshutdownmethod. If the config has a value, it will be set.
However, this issue can happen to any bot relying on default values and modifying their config. Probably it should be later solved on the IntelMQ library level.