mirror of
https://github.com/ZhuLinsen/daily_stock_analysis.git
synced 2026-10-06 14:43:29 +08:00
main
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Wenyu Chiouandzhulinsen
|
0f57e4e2e5 |
fix(data): parse yfinance dividends from the single-column DataFrame yfinance 1.2.x returns (#1867)
yfinance 1.2.x returns Ticker.dividends as a single-column DataFrame instead of a Series. YfinanceFundamentalAdapter iterated it with `.items()`, which for a DataFrame yields (column_name, Series) — so `_safe_float(Series)` returned None, every dividend event was dropped, and the TTM cash/count silently fell back to `info.trailingAnnualDividendRate` (e.g. 24.0 with "0 次") instead of the true TTM sum. Coerce to a Series before iterating. Affects every yfinance-backed market (US/HK/JP/KR/TW); surfaced on a live TW report (2330.TW showed 24.0 / 0 payouts vs the real ~22 across 4). + a regression test that feeds a single-column DataFrame and asserts the events + TTM sum are parsed. Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com> |
||
|
Wenyu Chiou
|
9433f610cc |
fix(market-phase): detect the tw closing auction (13:25-13:30) (#1869)
_CLOSING_AUCTION_WINDOW_MINUTES had no tw entry, so `.get(market, 0)` gave a
zero-width window and infer_market_phase("tw") could never return CLOSING_AUCTION:
the last tick before 13:30 stayed INTRADAY and 13:30 flipped straight to POSTMARKET.
TWSE/TPEx run a 5-minute closing call auction (13:25-13:30), so add "tw": 5.
+ a phase-boundary regression test (no lunch break, 13:30 half-hour close):
13:24 INTRADAY / 13:25-13:29 CLOSING_AUCTION / 13:30 POSTMARKET.
|
||
|
Wenyu Chiouandzhulinsen
|
7230465da5 |
feat(market): v2.1 — surface tw 三大法人 in report + LLM prompt, fix TWD currency (Refs #1777) (#1866)
* feat(market): v2.1 — consume tw 三大法人 in report + LLM prompt, fix TWD currency label The merged 三大法人 (institutional flows) data (#1829/#1841/#1855/#1863) reached get_fundamental_context but was never surfaced: the report renderer and the LLM prompt-builder never read the institution block, so a real analysis run showed a tw report with no 三大法人 figures. This wires the last mile (tw-only, additive, fail-open) and fixes a TWD currency mislabel found in the same run: - report render (src/notification.py): _append_institutional_flow renders a 三大法人 net-buy/sell table (外资/投信/自营/合计 in 万股/亿股) when the institution block is status='ok'; _get_fundamental_blocks now extracts it. + zh/en/ko labels. - LLM prompt (src/analyzer.py): _format_prompt injects a 三大法人 section (raw net figures + a qualitative 台股筹码过滤器 note, mirroring the capital_flow section) when status='ok' and all four core nets are present. - fetch availability (data_provider/base.py): the tw institution fetch (a ~4-5s whole-market download) now uses the remaining stage budget instead of the ~3s per-symbol fetch cap that starved it and made the first/only stock of a run coin-flip to not_supported. Still bounded by the stage deadline (fail-open). - currency (src/notification.py): _CURRENCY_SUFFIX gains TWD -> 新台币 so TWD amounts (revenue/profit/dividend) no longer render as the A-share default 元 (reads as RMB). Verified end-to-end via a real analysis run: the tw report now shows the 三大法人 table and 新台币-labelled financials, and institution is reliably 'ok' on a single-stock run. Strictly additive — cn/hk/us/jp/kr byte-identical (gated on market=='tw' / status=='ok'); no capital_flow_signal / scoring / schema. + 11 tests. Dual review (code-reviewer APPROVE + silent-failure-hunter): the hunter caught a prompt-gate asymmetry (only total_net checked vs all four in the render gate); tightened to require all four core nets, matching the render/base.py gate. * fix(market): honour FUNDAMENTAL_FETCH_TIMEOUT_SECONDS=0 for the tw institution fetch Review on #1866: the v2.1 institution-timeout change (use the remaining stage budget instead of the ~3s per-fetch cap) inadvertently bypassed the existing FUNDAMENTAL_FETCH_TIMEOUT_SECONDS=0 semantic — with fetch_timeout=0, valuation and bundle are disabled (they gate on fetch_timeout) but the institution fetch still ran. Gate the institution fetch on `fetch_timeout > 0` as well, so fetch_timeout=0 disables it like the other fundamental fetches; when enabled it still uses the remaining stage budget (the whole-market download needs more than the per-symbol cap). + a regression test asserting fetch_timeout=0 -> institution not_supported and the fetcher is not called. --------- Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com> |
||
|
Wenyu Chiou
|
b889523396 |
feat(market): harden tw institutional fetcher — circuit breaker + TPEx date guard (#1864)
Two tw-only, strictly-additive, fail-open hardening changes to TwInstitutionalFetcher (benchmarked against docs/data-source-stability.md): - C1 circuit breaker: reuse data_provider.realtime_types.CircuitBreaker (the same class DataFetcherManager uses), keyed per market (twse/tpex), 3-fail / ~5-min cooldown / half-open probe. When an endpoint is unreachable the fetch is skipped fast and fails open, instead of paying timeout + throttle on every stock during an outage. The breaker tracks REACHABILITY: a hard network/HTTP error trips it; an empty / stat!=OK body still means the endpoint responded, so it counts as success (resets the failure streak and closes the breaker during recovery -- a no-data day mid-recovery can never strand it open). - C2 TPEx date guard: TPEx OpenAPI serves only the latest trading day; an explicit date that does not match the served record now fails open (None) instead of a wrong-day record. Tests: +6 (breaker opens after 3 failures & skips the 4th fetch; recovers after cooldown; 3 empties don't trip it; TPEx date match / mismatch / no-date). + CHANGELOG. Dual review (code-reviewer APPROVE + silent-failure-hunter): the hunter flagged a HALF_OPEN-recovery edge in the shared CircuitBreaker; resolved tw-side by treating an empty response as success (reachability), without modifying the shared class. |
||
|
Wenyu Chiou
|
b939f7579f |
feat(market): surface tw 三大法人 in the tw report institution block (Refs #1777) (#1863)
* feat(market): surface tw 三大法人 (institutional flows) in the tw report institution block Wire the merged TwInstitutionalFetcher (#1829/#1841/#1855) into the tw analysis report: in _build_offshore_fundamental_context, for market == "tw" only, the institution block now carries the raw 三大法人 net buy/sell figures (foreign / trust / dealer / total, unit=shares) from TWSE T86 / TPEx instead of the hardcoded not_supported. v2 scope confirmed by the maintainer on issue #1777. - tw-only + strictly additive: cn/hk/us/jp/kr offshore flows are byte-identical (the not_supported loop just excludes institution; non-tw still gets the same not_supported block). Pinned by tests for us/hk/jp/kr. - fail-open + default-on: any error or no-data -> not_supported, never interrupts the analysis. Wiring (import/construct) failures log at error (visible) but still fail open; fetch failures log at warning. - status "ok" only when all four core nets are present (a genuine 0 is kept). - raw figures only: no capital_flow_signal / scoring / weight / schema / Web. Tests: tests/test_tw_institution_report_wiring.py (9) covers data->ok, genuine-zero, None/raise/init-raise fail-open, missing-core-net->not_supported, us/hk/jp/kr byte-identical + fetcher-unused, and no-derived-key. + CHANGELOG. Dual review (code-reviewer + silent-failure-hunter): the hunter caught the combined try swallowing wiring (import/construct) errors silently at info -> split into loud-but-fail-open (error log + still not_supported, honoring #1777's never-interrupt rule) and guarded "ok" against a null core net. * fix(market): address Codex review on tw institution wiring (#1863) — stage timeout, overall status, docs - P1: run the tw institution fetch under the fundamental stage/fetch budget via _run_with_retry (like the other offshore blocks), so a slow / rate-limited TWSE/TPEx call fails open at the deadline instead of pushing the analysis past it. - P2: count a present tw institution toward the OVERALL fundamental status so a report with only 三大法人 data still surfaces fundamentals (consumers key off the top-level status). missing_fields keeps the original three blocks, so offshore markets without institution data stay byte-identical. - P2 (docs): sync docs/market-support.md (drop the now-false "data-layer only / not wired into report" clause) + add a tw institution note to docs/full-guide.md and docs/full-guide_EN.md (AGENTS.md requires user-visible report changes to update guides). Tests: + stage-timeout (a 2s fetch is abandoned at the ~0.3s budget, fail-open) + overall-status surfacing assertion. 10 tw-wiring tests + offshore/tw/jp-kr suites green. |
||
|
Wenyu Chiouandzhulinsen
|
e80038009b |
test(tw): live-network drift smoke + network-marked tests for 三大法人 fetcher (#1855)
* test(tw): add live-network drift smoke + network-marked tests for 三大法人 fetcher The offline tests (tests/test_tw_institutional_fetcher.py) pin the parser to frozen fixtures, so they cannot notice an upstream TWSE T86 / TPEx feed change. Add two additive, tw-only drift detectors that hit the real public endpoints: - tests/tw_institutional_live_smoke.py: manual non-pytest smoke (mirrors tests/longbridge_live_smoke.py). Checks endpoint liveness + core-column-name presence, and cross-checks the fetcher's foreign/trust/dealer/total against the raw columns plus the always-true reconstruction total == foreign + foreign_dealer + trust + dealer (the 3-term identity only holds when the foreign-dealer sub-component is 0). - tests/test_tw_institutional_network.py: @pytest.mark.network, run only by the non-blocking Network Smoke cron (pytest -m network); the blocking backend gate runs pytest -m "not network" (scripts/ci_gate.sh) so these never gate a PR. Two self-contained tests cross-check the fetcher against the raw feed. No production code touched (data_provider/base.py unchanged); fail-open and no-silent-pass preserved. A renamed core/foreign-dealer column or a non-JSON response (maintenance page / URL migration) is reported LOUD as drift; only a transport error or non-trading-day soft-skips. Verified live against today's feeds and via a negative-path simulation of every drift/transient case. Dual review (code-reviewer APPROVE + silent-failure-hunter PROCEED) caught and fixed soft-skip paths that had masked feed drift (non-JSON body swallowed as a blip; foreign-dealer rename fabricated via `or 0`). * test(tw): fail loud when a stock is present in the raw feed but the fetcher returns None Addresses the review correctness-blocker on #1855. The TWSE/TPEx drift tests called the fetcher first and soft-skipped on a None result as "transient / suspended" WITHOUT checking the raw feed — so an upstream parse-prerequisite drift (e.g. a 民國->ISO date-format change that _parse_tpex_row / _parse_t86_row cannot convert) would make get_institutional_net() fail-open to None and be silently skipped, the exact fail-open these tests exist to catch (and contrary to the PR's stated "民國->ISO date switch fails loud"). Reorder both the smoke and the network tests: resolve the target stock's raw row FIRST, then if the fetcher returns None — fail LOUD when the row IS present in the raw feed (parse/date drift), and only soft-skip when the row is genuinely absent (non-trading day / suspended / transient). Verified: a negative-path simulation (raw row present + an unconvertible payload date) now reports drift and fails, the live happy path still passes, and the network tests are stable across repeated runs (transient blips hit both the raw fetch and the fetcher together -> consistent skip, never a false fail). --------- Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com> |
||
|
Wenyu Chiouandzhulinsen
|
2a65384654 |
perf(market): coalesce concurrent tw-institutional fetches (cache-stampede guard) (#1841)
Hardens the merged #1777 TwInstitutionalFetcher. Under concurrent callers for the same (market, date) the previous check-then-fetch could issue N duplicate upstream requests -- which for the TWSE T86 RWD endpoint risks tripping its informal ~3 req/5s ban. A per-(market,date) in-flight lock now coalesces same-key callers into a single fetch (double-checked against the cache); different keys still fetch in parallel, and the master lock is never held across network I/O. - data_provider/tw_institutional_fetcher.py: per-key in-flight lock + _read_cache / _key_lock helpers. Behavior is otherwise unchanged -- still fail-open, still caches only non-empty results, tw-only, no data_provider/base.py change. - tests: + concurrent-same-key-coalesces-to-single-fetch (8 threads -> 1 request), different-keys-not-coalesced, and HTTP-error (429) fail-open - docs/CHANGELOG.md: [改进] entry Refs #1777 Co-authored-by: zhulinsen <42829555+ZhuLinsen@users.noreply.github.com> |
||
|
Wenyu Chiou
|
7715896fe4 |
feat(market): tw institutional-flows (三大法人) data-layer fetcher (#1829)
Implements the #1777 maintainer-greenlit Phase-2 data layer: a self-contained, tw-only fetcher for Taiwan per-stock institutional (外資/投信/自營商) net buy/sell. Strictly additive -- no change to the existing cn/hk/us/jp/kr flows in data_provider/base.py, and not yet wired into the report/Web/scoring path (a deliberate follow-up per the maintainer's scope). - data_provider/tw_institutional_fetcher.py (NEW): TwInstitutionalFetcher - 上市 .TW -> TWSE T86 legacy rwd JSON endpoint (西元 date, comma values) - 上櫃 .TWO -> TPEx OpenAPI tpex_3insti_daily_trading (民國 date, plain ints) - T86 columns are read by NAME (validated against the payload `fields` header), so a TWSE column rename / reorder fails open instead of silently shipping misaligned numbers under stale indices - foreign_net = 外陸資 (ex 外資自營商, T86) / dealer-excluded foreign (TPEx) so the breakdown matches the official 三大法人 total; total_net is the official figure; unit = shares, signs preserved - whole-market single-day cache keyed by (market, date), filtered per stock; ~3 req/5s throttle (own lock) for the T86 endpoint - fail-open: any network/rate-limit/empty/unknown-stock returns None; a missing or renamed column drops the row (never a fabricated 0); a row whose trading date cannot be attributed (TPEx 民國 unconvertible) is dropped; empty/failed fetches are not cached (no TTL-long blackout) - tests/test_tw_institutional_fetcher.py (NEW): 20 offline tests with fixtures trimmed from real T86 (2330) / TPEx (3105) responses; pins the net breakdown + sign, 民國->西元 conversion, routing, caching, and fail-open -- including column reorder (parsed by name), column rename / missing header (fail-open), the missing-column-vs-genuine-zero distinction, and unconvertible TPEx dates - docs/market-support.md + docs/CHANGELOG.md: data-source capability boundary + OGDL v1 license note; no new config (.env.example untouched) Addresses the #1829 review (read T86 by field name; drop TPEx rows with an unconvertible date; foreign_net excludes foreign-dealers). Sources are 政府開放資料 under 政府資料開放授權條款第 1 版 (OGDL v1, commercial-safe). Refs #1777 |
||
|
Wenyu Chiou
|
cb72be7408 |
feat(market): bring tw to first-class on decision-signal / portfolio / intelligence (service + API + frontend) (#1801)
Follow-up to the #1773 data-layer MVP (Taiwan suffix-only detection + routing,
merged in
|
||
|
Wenyu Chiou
|
2086e3cca4 |
feat(market): Taiwan (台股) suffix-only market detection + routing (#1773)
Per the maintainer review on #1772/#1773, scope this PR to market detection
and data routing only — no Taiwan stock-index / seed / pool-resolution, so a
bare code can never be silently rewritten to a .TW suffix (the index path is a
deferred follow-up with the jp/kr-style honest doc + entry-point tests).
- data_provider/base.py: _is_tw_market / _is_tw_suffix_stock, _market_tag,
normalize_stock_code, get_daily_data + get_realtime_quote routing,
_DAILY_MARKET_FETCHER_SUPPORT (YfinanceFetcher only), offshore fundamental path
- data_provider/yfinance_fetcher.py: _get_tw_main_indices (^TWII / ^TWOII) +
.TW/.TWO suffix passthrough
- src/core/trading_calendar.py: XTAI / Asia/Taipei + get_market_for_stock
- src/market_context.py: detect_market + Taiwan prompt guidelines
- src/{market_phase_summary, services/stock_code_utils, core/pipeline}: tw plumbing
(stock_code_utils only recognizes .TW/.TWO codes; no bare->suffix rewrite)
- src/services/decision_signal_extractor.py: gracefully skip tw (recognized by
the data layer but not yet supported by the signal service) — no swallowed
ValueError + traceback
- tests: test_tw_market_support.py + test_decision_signal_extractor.py
(test_build_payload_skips_tw_market_gracefully — real tw skip regression)
- docs/market-support.md + CHANGELOG
Deferred follow-ups: stock index/seed + autocomplete + frontend, API market
enums + api_spec, and the service-layer market enums.
Bare numeric codes keep A-share semantics; only an explicit .TW/.TWO suffix
opts into Taiwan.
Refs #1772
|