-
Notifications
You must be signed in to change notification settings - Fork 518
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
config/index does not always preserve index correctly #957
Comments
For me or has always been one of the golden Riemann rules: thou shalt only use one index. It's stated in the documentation pretty clearly, don't you think? |
I must admit that I don't remember reading that in the documentation, but memory is not my best talent! :) Do you know where it is? The API documentation is the stuff that I always have open when I'm working with Riemann, so I might add that warning to the docstring if we need it (similar to the one that But I think we can do more, and ask whether that rule should be necessary. On my team, we have another rule. It's: Thou shalt not write an s-expression that fits not on thy 4K monitor. If we can only call Currently, the behaviour of (instrumentation {:enabled? false})
(tcp-server {:host "0.0.0.0"})
(periodically-expire 1)
(info "Let binding index.")
(let [index (index)]
(streams
(not-expired index prn)
(expired prn)))
(info "Calling (index).")
(index) works correctly if you send Riemann a A couple of ideas:
These options are presented in descending order of my personal preference, but also in descending order of invasiveness. What do folks think about them (or others)? Cheers. |
Fix the "orphaned indicies" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load.
Fix the "orphaned indicies" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load. Closes: riemann#957
Fix the "orphaned indicies" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load. Closes: riemann#957
I ended up going with option 1, which is implemented in #997. |
Set current index on first call of config/index Fix the "orphaned indices" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load.
The
index
function inconfig.clj
doesn't always preserve the existing index as expected. In particular, whenindex
is called multiple times within the same config, the index is replaced, which results in odd behaviour.Given this config:
and adding some debug logs to the index function, we see this output:
This causes weird behaviour with the index in any config where
index
is called (probably let-bound) multiple times. In particular, expiry gets messed up. With this config:when injecting an event with
:ttl 1
, we'd expect to print an expired event within 2 seconds, but it never arrives:Is this because the reaper is attached to a different index than the one that contains the event?
Partly, I think this is related to the behaviour of
index
vs its docstring:"Set the index used by this core. Returns the index."
It looks like it technically sets the index used by the next core, and that change is not made concrete until something or someone calls for a core transition. Adding some more debug, it looks like
(:index @core)
remainsnil
even after several calls to(index)
:A work-around in the config is to manually "fix" the index into the core by calling a combination of
index
andapply!
before doing anything serious:This config preserves the index for future calls to
index
, and expires the test event as expected:I wonder if the
index
function itself should ensure that the index is fixed on startup like this. Alternatively, should there be an additionalindex!
function that really does set the index on the current core?So, please excuse the wall-o-text. I'd love to hear any ideas from experienced Riemann hackers on whether this looks like a real problem, and if so, what we could (safely) do to change it.
Thank you.
The text was updated successfully, but these errors were encountered: