wol/radar
1
0
Fork 0
mirror of https://gitee.com/freshday/radar.git synced 2025-12-26 07:16:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #31 from wfh45678/develop

Browse Source 
1.query and hit report
2.machine learning
This commit is contained in:
feihu.wang 2020-01-19 09:23:44 +08:00 committed by GitHub
commit 03b34b7ee1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
80 changed files with 3023 additions and 371 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
README.md
View File

@ -1,6 +1,6 @@
# 风控引擎（Radar）
## 项目介绍
一款基于java语言，使用Springboot + Mongodb + Groovy 等框架搭建的轻量级实时风控引擎，适用于反欺诈应用场景，极简的配置，真正做到了开箱即用。
一款基于java语言，使用Springboot + Mongodb + Groovy + Es等框架搭建的轻量级实时风控引擎，适用于反欺诈应用场景，极简的配置，真正做到了开箱即用。
通过学习本项目能快速了解**风险的定义**，进而**量化风险** ，最后达到**集中管理风险**的目的。
A real-time risk analysis engine,which can update risk rule in real-time and make it effective immediately.
It applies to the anti-fraud application perfectly.
@ -17,8 +17,8 @@ The project code called Radar, like the code, monitor the transaction at the bac
* 配置简单，开箱即用！
## 相关站点
Github: https://github.com/wfh45678/radar
Gitee: https://gitee.com/freshday/radar // 码云为镜像网站，贡献代码请提交到 github
Github: https://github.com/wfh45678/radar
官网： http://radar.pgmmer.top
Wiki: https://gitee.com/freshday/radar/wikis/home
@ -27,9 +27,10 @@ The project code called Radar, like the code, monitor the transaction at the bac
伴随着移动互联网的高速发展，羊毛党快速崛起，从一平台到另一个平台，所过之处一地鸡毛，这还不是最可怕的，
随之而来的黑产令大部分互联网应用为之胆寒，通常新上线的APP的福利比较大，风控系统不完善，BUG 被发现的频率也比较高，
黑产利用BUG短时间给平台带来了巨大的损失，某多多的（100元测试优惠券，一夜损失上百万W）就是一例。
针对这一现象， 拥有一款实时的风控引擎是所有带有金融性质的APP 的当务之急，
Radar 应景而生，Radar本来是笔者前公司的一个内部项目，公司现在不复存在，考虑到项目本身的价值，
现在使用Springboot进行改造，并删除了很多本地化功能，只保留风控引擎核心，更具通用型，二次开发成本低。
针对这一现象， 拥有一款实时的风控引擎是所有带有金融性质的APP 的当务之急，Radar应景而生。
Radar前身是笔者前公司的一个内部研究项目，由于众多原因项目商业化失败，考虑到项目本身的价值，弃之可惜，
现使用Springboot进行重构，删除了很多本地化功能，只保留风控引擎核心，更加通用，更加轻量，二次开发成本低，
开源出来，希望能给有风控需求的你们带来一些帮助。
## 项目初衷
我们知道企业做大后，会有很多产品线，而几乎每一个产品都需要做风险控制，通常我们都是把风险控制的逻辑写在相应的业务功能代码里，
@ -42,7 +43,7 @@ The project code called Radar, like the code, monitor the transaction at the bac
前后端分离架构
后端技术框架： SpringBoot + Mybatis + tkMapper + Mysql + MongoDB + Redis + Groovy + Swagger
后端技术框架： SpringBoot + Mybatis + tkMapper + Mysql + MongoDB + Redis + Groovy + ES + Swagger
前端技术框架： React(SPA)
@ -56,9 +57,11 @@ The project code called Radar, like the code, monitor the transaction at the bac
* Mysql ： 本项目中关系数据库，主要用于存放 风险模型的元信息。
* MongoDB： 用于存放事件JSON， 提供基本统计学计算（例如：max, min, sum, avg, ），
* MongoDB： 用于存放事件JSON， 提供基本统计学计算（例如：max, min, sum, avg,），
复杂的统计学概念（sd,variance, etc...）在内存中计算。
* ES： 提供数据查询和规则命中报表服务。
* Redis： 提供缓存支持，Engine 利用发布订阅特性监听管理端相关配置的更新
* Groovy： 规则引擎，风控规则最后都生成 groovy 脚本， 实时编辑，动态生成，即时生效。
@ -66,11 +69,11 @@ The project code called Radar, like the code, monitor the transaction at the bac
* Swagger: Rest API 管理
## 名词解释
### Model: 模型
用户行为事件， 例如：注册，登录，购买，提现。。。
### 模型三要素
也就是事件行为三要素，风控系统的核心定义：事件流水ID(例如：交易流水号)，实体ID（例如：userId），事件发生时间（例如：交易时间）,
简单来说就是谁什么时候做了什么事。（who, when, what）
### Model: 事件模型
用户行为事件， 例如：注册，登录，购买，提现等具体的业务行为。
### 事件模型三要素
也就是事件行为三要素，风控系统的核心定义：事件流水ID(例如：交易流水号)，实体ID（例如：userId,表示某个人或物），事件发生时间（例如：交易时间）,
简单来说就是谁（可以是人，也可以是物）什么时候做了什么事。（who, when, what）
### PreItem: 预处理
像经纬度，IP，手机号码段等事件属性，可能无法直接计算，通过预处理插件 转换成 其他格式，
例如:经纬度，ip 可以通过对应插件变成位置和地址，手机号码可以通过插件获取其它系统的用户画像信息等。
@ -90,9 +93,9 @@ The project code called Radar, like the code, monitor the transaction at the bac
---
## [使用手册](https://gitee.com/freshday/radar/wikis/manual?sort_id=1637446)
使用手册里面有大量的图片，为了方便使用，故推荐码云的wiki 链接,
https://gitee.com/freshday/radar/wikis/manual?sort_id=1637446
## [使用手册](https://gitee.com/freshday/radar/wikis/manual)
使用手册里面有大量的图片，为了方便国内用户使用，故推荐码云的wiki 链接,
https://gitee.com/freshday/radar/wikis/manual
## 演示入口
@ -104,7 +107,7 @@ https://gitee.com/freshday/radar/wikis/manual?sort_id=1637446
建议大家自行注册用户，避免使用同样的测试账号受干扰.
## 未完待续
[Release Note:](https://gitee.com/freshday/radar/wikis/release%20note?sort_id=1723765) https://gitee.com/freshday/radar/wikis/release%20note
### 重大特性
* 支持机器学习
* 数据分析平台
@ -112,11 +115,12 @@ https://gitee.com/freshday/radar/wikis/manual?sort_id=1637446
## 致谢
感恩 XWF 团队，感谢参入的每一位小伙伴，后续征得同意后会一一列出名字。
千面怪, 烈日下的从容, DerekDingLu, king, sanying2012, 紫泉夜, 玄梦
成书平, 徐帅...
成书平, 徐帅，郭锐 ...
## Contact to
如果喜欢本项目，Star支持一下, 让更多人了解本项目，谢谢！
独乐乐不如众乐乐，微信扫码或手动（nicedream7758）加群一起嗨！
![微信交流群](http://radar.pgmmer.top/radar/wx2.jpg)
提示：进群需要捐赠，0.1元起
![微信交流群](http://radar.pgmmer.top/radar/wx2.jpg)
Copyright © WFH
Copyright © WangFeiHu 2020

175
pom.xml
View File

@ -19,14 +19,20 @@
</parent>
<groupId>com.pgmmers</groupId>
<artifactId>radar</artifactId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
<name>radar</name>
<description>Demo project for Spring Boot</description>
<packaging>pom</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<mapper.version>4.0.0</mapper.version>
<mybatisGenerator.version>1.3.7</mybatisGenerator.version>
<mysql.version>5.1.47</mysql.version>
<springboot.version>2.1.7.RELEASE</springboot.version>
<tomcat.version>8.5.37</tomcat.version>
<tensorflow.version>1.12.0</tensorflow.version>
</properties>
@ -35,7 +41,7 @@
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
@ -43,41 +49,36 @@
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mybatis.generator</groupId>
<artifactId>mybatis-generator-core</artifactId>
<version>1.3.7</version>
<version>${mybatisGenerator.version}</version>
</dependency>
<dependency>
<groupId>com.github.pagehelper</groupId>
<artifactId>pagehelper</artifactId>
<version>5.1.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/tk.mybatis/mapper -->
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
<version>4.0.0</version>
<version>${mapper.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/tk.mybatis/mapper-spring-boot-starter -->
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper-spring-boot-starter</artifactId>
@ -86,13 +87,20 @@
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>3.9</version>
<artifactId>poi-ooxml</artifactId>
<version>3.17</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>3.9</version>
<artifactId>poi</artifactId>
<version>3.17</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml-schemas</artifactId>
<version>3.17</version>
</dependency>
<dependency>
@ -119,40 +127,51 @@
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.8.5</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.3.5</version>
<version>6.8.5</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<groupId>org.hdrhistogram</groupId>
<artifactId>HdrHistogram</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>transport-netty4-client</artifactId>
<version>6.8.5</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>2.1.7.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>2.1.10.RELEASE</version>
<version>${springboot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
</dependency>
<dependency>
@ -164,13 +183,13 @@
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.47</version>
<version>${mysql.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
<version>2.1.7.RELEASE</version>
<version>${springboot.version}</version>
</dependency>
<dependency>
@ -202,11 +221,103 @@
<version>28.0-jre</version>
</dependency>
<!-- admin 模块独立引用-->
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt</artifactId>
<version>0.9.1</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<!-- commons 模块独立引用-->
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.4.6</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<!-- dal 模块独立引用
（升级项目时请注意此处版本冲突，
mapper-spring-boot-starter、
mybatis-spring-boot-starter 引入的pagehelper-spring-boot-starter 均引入此包）-->
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>3.4.6</version>
</dependency>
<dependency>
<groupId>org.tensorflow</groupId>
<artifactId>tensorflow</artifactId>
<version>${tensorflow.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${springboot.version}</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.mybatis.generator</groupId>
<artifactId>mybatis-generator-maven-plugin</artifactId>
<version>${mybatisGenerator.version}</version>
<configuration>
<configurationFile>
${basedir}/src/main/resources/generator/generatorConfig.xml
</configurationFile>
<overwrite>true</overwrite>
<verbose>true</verbose>
</configuration>
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.version}</version>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
<version>${mapper.version}</version>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
<artifactId>radar-commons</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>

79
radar-admin/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>radar-admin</artifactId>
@ -21,10 +21,6 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
@ -33,43 +29,21 @@
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/tk.mybatis/mapper -->
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/tk.mybatis/mapper-spring-boot-starter -->
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<!-- ThymeLeaf 依赖 -->
<dependency>
<groupId>org.springframework.boot</groupId>
@ -79,61 +53,40 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
<artifactId>radar-service</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
<artifactId>radar-service-impl</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.github.pagehelper</groupId>
<artifactId>pagehelper-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>com.github.pagehelper</groupId>
<artifactId>pagehelper</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt</artifactId>
<version>0.9.1</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>easyexcel</artifactId>
<version>2.1.3</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
</dependency>
</dependencies>
<build>
@ -142,14 +95,6 @@
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.1.7.RELEASE</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

6
radar-admin/src/main/java/com/pgmmers/AdminApplication.java
View File

@ -2,13 +2,17 @@ package com.pgmmers;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.data.mongo.MongoDataAutoConfiguration;
import org.springframework.boot.autoconfigure.elasticsearch.rest.RestClientAutoConfiguration;
import org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration;
import tk.mybatis.spring.annotation.MapperScan;
@SpringBootApplication
@SpringBootApplication(exclude = {MongoAutoConfiguration.class, MongoDataAutoConfiguration.class, RestClientAutoConfiguration.class})
@MapperScan("com.pgmmers.radar.mapper")
public class AdminApplication
{
public static void main( String[] args ){
System.setProperty("es.set.netty.runtime.available.processors", "false");
SpringApplication.run(AdminApplication.class, args);
}
}

47
radar-admin/src/main/java/com/pgmmers/radar/config/EsConfig.java Normal file
View File

@ -0,0 +1,47 @@
package com.pgmmers.radar.config;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.net.InetAddress;
@Configuration
public class EsConfig {
private static final Logger logger = LoggerFactory.getLogger(EsConfig.class);
@Value("${elasticsearch.ip}")
private String hostName;
@Value("${elasticsearch.port}")
private String port;
@Value("${elasticsearch.cluster.name}")
private String clusterName;
@Value("${elasticsearch.pool-size}")
private String poolSize;
@Bean(name = "transportClient")
public TransportClient transportClient() {
logger.info("Elasticsearch初始化开始。。。。。");
TransportClient transportClient = null;
try {
// 配置信息
Settings esSetting = Settings.builder()
.put("cluster.name", clusterName) //集群名字
.put("client.transport.sniff", true)//增加嗅探机制，找到ES集群
.put("thread_pool.search.size", Integer.parseInt(poolSize))//增加线程池个数，暂时设为5
.build();
//配置信息Settings自定义
transportClient = new PreBuiltTransportClient(esSetting);
TransportAddress transportAddress = new TransportAddress(InetAddress.getByName(hostName), Integer.valueOf(port));
transportClient.addTransportAddresses(transportAddress);
} catch (Exception e) {
logger.error("elasticsearch TransportClient create error!!", e);
}
return transportClient;
}
}

8
radar-admin/src/main/java/com/pgmmers/radar/config/RedisConfig.java
View File

@ -1,8 +1,5 @@
package com.pgmmers.radar.config;
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CachingConfigurerSupport;
import org.springframework.cache.annotation.EnableCaching;
@ -11,11 +8,6 @@ import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import javax.annotation.Resource;
import java.lang.reflect.Method;
import java.util.HashSet;

20
radar-admin/src/main/java/com/pgmmers/radar/config/RestTemplateConfig.java Normal file
View File

@ -0,0 +1,20 @@
package com.pgmmers.radar.config;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;
@Configuration
public class RestTemplateConfig {
@Autowired
private RestTemplateBuilder restTemplateBuilder;
@Bean
public RestTemplate getRestTemplate() {
return restTemplateBuilder.build();
}
}

16
radar-admin/src/main/java/com/pgmmers/radar/controller/ActivationApiController.java
View File

@ -99,6 +99,22 @@ public class ActivationApiController {
return result;
}
@GetMapping("/absColumns/{modelId}")
public CommonResult getAbstractionColumns(@PathVariable Long modelId) {
CommonResult result = new CommonResult();
result.setSuccess(true);
List<AbstractionVO> listAbstract = abstractionService.listAbstraction(modelId);
DataColumnInfo ds = new DataColumnInfo(DataType.ABSTRACTIONS.getDesc(), DataType.ABSTRACTIONS.getName());
if (listAbstract != null) {
for (AbstractionVO abs : listAbstract) {
ds.addChildren(abs.getLabel(), abs.getName(), FieldType.DOUBLE.name());
}
}
result.getData().put("columns", ds.getChildren());
return result;
}
@GetMapping("/rulecolumns/{modelId}")
public CommonResult getRuleColumns(@PathVariable Long modelId) {
List<DataColumnInfo> list = new ArrayList<>();

41
radar-admin/src/main/java/com/pgmmers/radar/controller/CommonApiController.java
View File

@ -1,21 +1,28 @@
package com.pgmmers.radar.controller;
import com.alibaba.excel.util.IoUtils;
import com.pgmmers.radar.enums.FieldType;
import com.pgmmers.radar.enums.PluginType;
import com.pgmmers.radar.service.common.CommonResult;
import com.pgmmers.radar.util.RandomValidateCode;
import com.pgmmers.radar.util.ZipUtils;
import com.pgmmers.radar.vo.common.PluginVO;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiParam;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
@ -28,6 +35,8 @@ public class CommonApiController {
public static Logger logger = LoggerFactory.getLogger(CommonApiController.class);
@Value("${sys.conf.workdir}")
public String workDir;
@GetMapping("/plugins")
public CommonResult plugins() {
@ -44,10 +53,10 @@ public class CommonApiController {
@GetMapping("/fieldtypes")
public CommonResult fieldTypes() {
CommonResult result = new CommonResult();
List<HashMap<String,Object>> fields = new ArrayList<HashMap<String,Object>>();
List<HashMap<String,Object>> fields = new ArrayList<>();
for (FieldType ft : FieldType.values()) {
//fields.add(ft.name());
HashMap<String,Object> map=new LinkedHashMap<String,Object>();
HashMap<String,Object> map=new LinkedHashMap<>();
map.put("name", ft.name());
map.put("desc", ft.getDesc());
fields.add(map);
@ -71,6 +80,26 @@ public class CommonApiController {
logger.error("get captcha error", e);
}
}
@PostMapping(value = "/upload")
public CommonResult upload(@ApiParam(value = "file") @RequestPart("file") MultipartFile file, @RequestParam(defaultValue = "") String key) {
CommonResult result = new CommonResult();
String fileName = file.getOriginalFilename();
try {
String path = workDir + "/" + fileName;
String decomposePath = path.substring(0, path.lastIndexOf("."));
FileOutputStream fos = new FileOutputStream(new File(workDir + "/" + fileName));
IoUtils.copy(file.getInputStream(), fos);
fos.flush();
fos.close();
if (!StringUtils.isEmpty(key) && key.equals("machine")) {
ZipUtils.unZipIt(path, decomposePath);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
}
return result;
}
}

2
radar-admin/src/main/java/com/pgmmers/radar/controller/EventApiController.java
View File

@ -75,7 +75,7 @@ public class EventApiController {
public CommonResult search(@RequestBody TermQuery term, HttpSession session) {
CommonResult result = new CommonResult();
PageResult<Object> page = new PageResult<>(1, 10, 0, new ArrayList<>());
//page = eventService.query(term);
page = eventService.query(term);
if (page != null) {
result.getData().put("page", page);
result.setSuccess(true);

73
radar-admin/src/main/java/com/pgmmers/radar/controller/ModelConfigApiController.java Normal file
View File

@ -0,0 +1,73 @@
package com.pgmmers.radar.controller;
import com.pgmmers.radar.service.common.CommonResult;
import com.pgmmers.radar.service.model.ModelConfService;
import com.pgmmers.radar.vo.model.ModelConfVO;
import io.swagger.annotations.Api;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.*;
/**
* 机器学习配置 api.
* @author wangfeihu
*/
@RestController
@RequestMapping("/services/v1/modelConfig")
@Api(value = "ModelConfigApi", description = "模型机器学习配置相关操作", tags = {"机器学习配置API"})
public class ModelConfigApiController {
@Autowired
private ModelConfService modelConfService;
@GetMapping("/list/{modelId}")
public CommonResult list(@PathVariable Long modelId) {
CommonResult result = new CommonResult();
ModelConfVO modelConfVO = modelConfService.getByModelId(modelId);
if (modelConfVO != null) {
result.getData().put("modelConfig", modelConfVO);
}
result.setSuccess(true);
return result;
}
@PutMapping
public CommonResult save(@RequestBody ModelConfVO modelConf) {
CommonResult result = new CommonResult();
if (modelConf.getId() == -1L) {
modelConf.setId(null);
}
if (!validate(modelConf)) {
result.setMsg("请按照提示输入必要字段！");
return result;
}
modelConfService.save(modelConf);
result.setSuccess(true);
return result;
}
/**
* 检验上传字段。
* @param modelConf
* @return
* @author wangfeihu
*/
private boolean validate(ModelConfVO modelConf) {
boolean result = true;
if (StringUtils.isEmpty(modelConf.getName())
|| StringUtils.isEmpty(modelConf.getOperation())
|| StringUtils.isEmpty(modelConf.getTag())
|| StringUtils.isEmpty(modelConf.getPath())) {
result = false;
}
if (modelConf.getId() == null) {
if ( StringUtils.isEmpty(modelConf.getConfParam().getFeed())
|| StringUtils.isEmpty(modelConf.getConfParam().getExpressions())) {
result = false;
}
}
return result;
}
}

46
radar-admin/src/main/java/com/pgmmers/radar/controller/ModelConfigParamApiController.java Normal file
View File

@ -0,0 +1,46 @@
package com.pgmmers.radar.controller;
import com.pgmmers.radar.service.common.CommonResult;
import com.pgmmers.radar.service.model.ModelConfParamService;
import com.pgmmers.radar.service.model.ModelConfService;
import com.pgmmers.radar.vo.model.ModelConfParamVO;
import com.pgmmers.radar.vo.model.ModelConfVO;
import io.swagger.annotations.Api;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
/**
* 机器学习配置 api.
* @author wangfeihu
*/
@RestController
@RequestMapping("/services/v1/modelConfigParam")
@Api(value = "ModelConfigApi", description = "模型机器学习配置相关操作", tags = {"机器学习配置API"})
public class ModelConfigParamApiController {
@Autowired
private ModelConfParamService modelParamService;
@GetMapping("/{id}")
public CommonResult get(@PathVariable Long id) {
CommonResult result = new CommonResult();
ModelConfParamVO paramVO = modelParamService.get(id);
result.getData().put("param", paramVO);
result.setSuccess(true);
return result;
}
@PutMapping
public CommonResult save(@RequestBody ModelConfParamVO modelConf) {
CommonResult result = new CommonResult();
if (modelConf.getId() == -1L) {
modelConf.setId(null);
}
modelParamService.save(modelConf);
result.setSuccess(true);
return result;
}
}

2
radar-admin/src/main/java/com/pgmmers/radar/controller/RuleApiController.java
View File

@ -47,8 +47,6 @@ public class RuleApiController {
@PutMapping
public CommonResult save(@RequestBody RuleVO rule, HttpServletRequest request) {
// HttpSession session = request.getSession();
// UserVO user = (UserVO) session.getAttribute("user");
return ruleService.save(rule, contextHolder.getContext().getUsername());
}

22
radar-admin/src/main/resources/application.yml
View File

@ -50,20 +50,28 @@ mongodb:
url: mongodb://localhost:27017/radar
mobile:
info:
path: D:/soft/moble_info.csv
path: D:/radar/moble_info.csv #手机号码归属地文件
ip2region:
db:
path: D:/soft/ip2region.db
elksearch:
server: localhost
path: D:/radar/ip2region.db #IP地址库文件
elasticsearch:
ip: localhost
port: 9300
pool-size: 5
cluster:
name: elasticsearch
url: http://localhost:9200
logging:
level:
root: info
com.pgmmers.radar: info
com.pgmmers.radar.mapper: debug
org.elasticsearch: info
sys:
conf:
app: admin
entity-duplicate-insert: false
mongo-restore-days: 93
app: admin # admin 或者 engine， 根据启动的项目名称进行选择
entity-duplicate-insert: false # 事件是否允许重复插入
mongo-restore-days: 93 # 事件保存时间，默认3个月
workdir: d:\\radar # 工作目录
server:
port: 8080

0
radar-admin/src/main/resources/static/index.3b7ff13c-1.css → radar-admin/src/main/resources/static/index.f4b2bdd4-1.css
View File

0
radar-admin/src/main/resources/static/index.3b7ff13c-2.css → radar-admin/src/main/resources/static/index.f4b2bdd4-2.css
View File

0
radar-admin/src/main/resources/static/index.3b7ff13c-3.css → radar-admin/src/main/resources/static/index.f4b2bdd4-3.css
View File

2
radar-admin/src/main/resources/static/index.html
View File

@ -1,3 +1,3 @@
<!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width,initial-scale=1"> <title>风控引擎管理平台</title> <link rel="shortcut icon" href="/images/anquan.png"> <!--[if lt IE 10]>
<script src="https://as.alipayobjects.com/g/component/??console-polyfill/0.2.2/index.js,es5-shim/4.5.7/es5-shim.min.js,es5-shim/4.5.7/es5-sham.min.js,html5shiv/3.7.2/html5shiv.min.js,media-match/2.0.2/media.match.min.js"></script>
<![endif]--> <link href="./index.3b7ff13c-1.css" rel="stylesheet"><link href="./index.3b7ff13c-2.css" rel="stylesheet"><link href="./index.3b7ff13c-3.css" rel="stylesheet"></head> <body> <div id="react-content"></div> <script type="text/javascript" src="./main.3b7ff13c.js"></script></body> </html>
<![endif]--> <link href="./index.f4b2bdd4-1.css" rel="stylesheet"><link href="./index.f4b2bdd4-2.css" rel="stylesheet"><link href="./index.f4b2bdd4-3.css" rel="stylesheet"></head> <body> <div id="react-content"></div> <script type="text/javascript" src="./main.f4b2bdd4.js"></script></body> </html>

10
radar-admin/src/main/resources/static/main.3b7ff13c.js → radar-admin/src/main/resources/static/main.f4b2bdd4.js
View File

File diff suppressed because one or more lines are too long

BIN
radar-admin/src/main/resources/static/res/dataRecord_temp.xlsx Normal file
View File

Binary file not shown.

17
radar-commons/pom.xml
View File

@ -6,7 +6,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -21,10 +21,6 @@
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
@ -33,32 +29,27 @@
<dependency>
<groupId>org.mybatis.generator</groupId>
<artifactId>mybatis-generator-core</artifactId>
<version>1.3.7</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.4.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.9</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
@ -67,12 +58,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</build>

22
radar-commons/src/main/java/com/pgmmers/radar/util/ExcelUtils.java
View File

@ -55,7 +55,7 @@ public class ExcelUtils {
// 创建第一种字体样式（用于列名）
f.setFontHeightInPoints((short) 10);
f.setColor(IndexedColors.BLACK.getIndex());
f.setBoldweight(Font.BOLDWEIGHT_BOLD);
// f.setBoldweight(Font.BOLDWEIGHT_BOLD);
// 创建第二种字体样式（用于值）
f2.setFontHeightInPoints((short) 10);
@ -67,19 +67,19 @@ public class ExcelUtils {
// 设置第一种单元格的样式（用于列名）
cs.setFont(f);
cs.setBorderLeft(CellStyle.BORDER_THIN);
cs.setBorderRight(CellStyle.BORDER_THIN);
cs.setBorderTop(CellStyle.BORDER_THIN);
cs.setBorderBottom(CellStyle.BORDER_THIN);
cs.setAlignment(CellStyle.ALIGN_CENTER);
// cs.setBorderLeft(CellStyle.BORDER_THIN);
// cs.setBorderRight(CellStyle.BORDER_THIN);
// cs.setBorderTop(CellStyle.BORDER_THIN);
// cs.setBorderBottom(CellStyle.BORDER_THIN);
// cs.setAlignment(CellStyle.ALIGN_CENTER);
// 设置第二种单元格的样式（用于值）
cs2.setFont(f2);
cs2.setBorderLeft(CellStyle.BORDER_THIN);
cs2.setBorderRight(CellStyle.BORDER_THIN);
cs2.setBorderTop(CellStyle.BORDER_THIN);
cs2.setBorderBottom(CellStyle.BORDER_THIN);
cs2.setAlignment(CellStyle.ALIGN_CENTER);
// cs2.setBorderLeft(CellStyle.BORDER_THIN);
// cs2.setBorderRight(CellStyle.BORDER_THIN);
// cs2.setBorderTop(CellStyle.BORDER_THIN);
// cs2.setBorderBottom(CellStyle.BORDER_THIN);
// cs2.setAlignment(CellStyle.ALIGN_CENTER);
//设置列名
for(int i=0;i<columnNames.length;i++){
Cell cell = row.createCell(i);

8
radar-commons/src/main/java/com/pgmmers/radar/util/GroovyScriptUtil.java
View File

@ -3,7 +3,6 @@ package com.pgmmers.radar.util;
import groovy.lang.GroovyClassLoader;
import groovy.lang.GroovyObject;
import org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor;
import java.util.HashMap;
import java.util.Map;
@ -58,4 +57,11 @@ public class GroovyScriptUtil {
return null;
}
}
/**
* 删除不在使用的脚本关联的groovy object, 不然内存有溢出风险。
*/
public static void removeInactiveScript(String script){
passedClassMap.remove(script.hashCode() + "");
}
}

145
radar-commons/src/main/java/com/pgmmers/radar/util/ZipUtils.java Normal file
View File

@ -0,0 +1,145 @@
package com.pgmmers.radar.util;
import java.io.*;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
public class ZipUtils {
public static void main(String[] args) {
String zipFilePath = "/Users/xxx/Documents/temp/v6.zip";
String outputFolder = "/Users/xxx/Documents/temp/res/";
System.out.println(new File(zipFilePath).exists());
unZipIt(zipFilePath, outputFolder);
//测试2
try {
upZipFile(new File(zipFilePath), outputFolder);
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 解压文件
*
* @param zipFilePath 解压文件路径
* @param outputFolder 输出解压文件路径
*/
public static void unZipIt(String zipFilePath, String outputFolder) {
byte[] buffer = new byte[1024];
File folder = new File(outputFolder);
if (!folder.exists()) {
folder.mkdir();
}
try {
//get the zip file content
ZipInputStream zis = new ZipInputStream(new FileInputStream(zipFilePath));
ZipEntry ze = zis.getNextEntry();
while (ze != null) {
String fileName = ze.getName();
File newFile = new File(outputFolder + File.separator + fileName);
System.out.println("file unzip : " + newFile.getAbsoluteFile());
//create all non exists folders
//else you will hit FileNotFoundException for compressed folder
//大部分网络上的源码，这里没有判断子目录
if (ze.isDirectory()) {
newFile.mkdirs();
} else {
new File(newFile.getParent()).mkdirs();
FileOutputStream fos = new FileOutputStream(newFile);
int len;
while ((len = zis.read(buffer)) != -1) {
fos.write(buffer, 0, len);
}
fos.close();
}
ze = zis.getNextEntry();
}
zis.closeEntry();
zis.close();
System.out.println("Done");
} catch (IOException e) {
e.printStackTrace();
}
}
public static void unzip(File source, String out) throws IOException {
try (ZipInputStream zis = new ZipInputStream(new FileInputStream(source))) {
ZipEntry entry = zis.getNextEntry();
while (entry != null) {
File file = new File(out, entry.getName());
if (entry.isDirectory()) {
file.mkdirs();
} else {
File parent = file.getParentFile();
if (!parent.exists()) {
parent.mkdirs();
}
try (BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(file))) {
byte[] buffer = new byte[Math.toIntExact(entry.getSize())];
int location;
while ((location = zis.read(buffer)) != -1) {
bos.write(buffer, 0, location);
}
}
}
entry = zis.getNextEntry();
}
}
}
/**
* 把所有文件都直接解压到指定目录(忽略子文件夹)
*
* @param zipFile
* @param folderPath
* @throws ZipException
* @throws IOException
*/
public static void upZipFile(File zipFile, String folderPath) throws ZipException, IOException {
File desDir = new File(folderPath);
if (!desDir.exists()) {
desDir.mkdirs();
}
ZipFile zf = new ZipFile(zipFile);
for (Enumeration<?> entries = zf.entries(); entries.hasMoreElements(); ) {
ZipEntry entry = ((ZipEntry) entries.nextElement());
InputStream in = zf.getInputStream(entry);
String str = folderPath;
// str = new String(str.getBytes("8859_1"), "GB2312");
File desFile = new File(str, java.net.URLEncoder.encode(entry.getName(), "UTF-8"));
if (!desFile.exists()) {
File fileParentDir = desFile.getParentFile();
if (!fileParentDir.exists()) {
fileParentDir.mkdirs();
}
}
OutputStream out = new FileOutputStream(desFile);
byte buffer[] = new byte[1024 * 1024];
int realLength = in.read(buffer);
while (realLength != -1) {
out.write(buffer, 0, realLength);
realLength = in.read(buffer);
}
out.close();
in.close();
}
}
}

24
radar-dal/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -18,23 +18,10 @@
<artifactId>radar-dao</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>3.4.6</version>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
</dependency>
<dependency>
<groupId>com.github.pagehelper</groupId>
<artifactId>pagehelper</artifactId>
<artifactId>pagehelper-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
@ -44,18 +31,11 @@
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</build>

16
radar-dal/src/main/java/com.pgmmers.radar/dal/model/ModelConfDal.java Normal file
View File

@ -0,0 +1,16 @@
package com.pgmmers.radar.dal.model;
import com.pgmmers.radar.vo.model.ModelConfParamVO;
import com.pgmmers.radar.vo.model.ModelConfVO;
public interface ModelConfDal {
ModelConfVO get(Long id);
ModelConfVO getByModelId(Long modelId);
ModelConfVO save(ModelConfVO confVO);
ModelConfParamVO getParamById(Long id);
ModelConfParamVO saveParam(ModelConfParamVO paramVO);
}

106
radar-dal/src/main/java/com.pgmmers.radar/dal/model/impl/ModelConfDalImpl.java Normal file
View File

@ -0,0 +1,106 @@
package com.pgmmers.radar.dal.model.impl;
import com.pgmmers.radar.dal.model.ModelConfDal;
import com.pgmmers.radar.mapper.ModelConfMapper;
import com.pgmmers.radar.mapper.ModelConfParamMapper;
import com.pgmmers.radar.model.ModelConfPO;
import com.pgmmers.radar.model.ModelConfParamPO;
import com.pgmmers.radar.vo.model.ModelConfParamVO;
import com.pgmmers.radar.vo.model.ModelConfVO;
import org.springframework.beans.BeanUtils;
import org.springframework.stereotype.Service;
import tk.mybatis.mapper.entity.Example;
import javax.annotation.Resource;
import java.util.Date;
import java.util.List;
import java.util.stream.Collectors;
@Service
public class ModelConfDalImpl implements ModelConfDal {
@Resource
private ModelConfMapper modelConfMapper;
@Resource
private ModelConfParamMapper modelConfParamMapper;
@Override
public ModelConfVO get(Long id) {
ModelConfPO modelConfPO = modelConfMapper.selectByPrimaryKey(id);
return convert(modelConfPO);
}
@Override
public ModelConfVO getByModelId(Long modelId) {
Example example = new Example(ModelConfPO.class);
example.createCriteria().andEqualTo("modelId", modelId);
ModelConfPO modelConfPO = modelConfMapper.selectOneByExample(example);
return convert(modelConfPO);
}
@Override
public ModelConfVO save(ModelConfVO confVO) {
ModelConfPO conf = new ModelConfPO();
BeanUtils.copyProperties(confVO, conf);
if (conf.getId() == null) {
conf.setCreateTime(new Date());
conf.setUpdateDate(new Date());
modelConfMapper.insert(conf);
confVO.setId(conf.getId());
ModelConfParamPO paramPO = new ModelConfParamPO();
paramPO.setFeed(confVO.getConfParam().getFeed());
paramPO.setExpressions(confVO.getConfParam().getExpressions());
paramPO.setMoldId(conf.getId());
modelConfParamMapper.insert(paramPO);
confVO.getConfParam().setId(paramPO.getId());
} else {
modelConfMapper.updateByPrimaryKeySelective(conf);
}
return confVO;
}
@Override
public ModelConfParamVO getParamById(Long id) {
ModelConfParamVO modelConfParamVO = new ModelConfParamVO();
ModelConfParamPO po = modelConfParamMapper.selectByPrimaryKey(id);
BeanUtils.copyProperties(po,modelConfParamVO);
return modelConfParamVO;
}
@Override
public ModelConfParamVO saveParam(ModelConfParamVO paramVO) {
ModelConfParamPO po = new ModelConfParamPO();
BeanUtils.copyProperties(paramVO, po);
if (po.getId() == null) {
modelConfParamMapper.insert(po);
paramVO.setId(po.getId());
} else {
modelConfParamMapper.updateByPrimaryKeySelective(po);
}
return paramVO;
}
private ModelConfVO convert(ModelConfPO modelConfPO) {
ModelConfVO vo = null;
if (modelConfPO != null) {
vo = new ModelConfVO();
BeanUtils.copyProperties(modelConfPO, vo);
fitParams(vo);
}
return vo;
}
private void fitParams(ModelConfVO mold) {
if (mold != null) {
Example example = new Example(ModelConfParamPO.class);
example.createCriteria().andEqualTo("moldId", mold.getId());
List<ModelConfParamPO> moldParamList = modelConfParamMapper.selectByExample(example);
List<ModelConfParamVO> list = moldParamList.stream().map(modelConfParamPO -> {
ModelConfParamVO modelConfParamVO = new ModelConfParamVO();
BeanUtils.copyProperties(modelConfParamPO, modelConfParamVO);
return modelConfParamVO;
}).collect(Collectors.toList());
mold.setParams(list);
}
}
}

2
radar-dal/src/main/java/com.pgmmers.radar/dal/model/impl/RuleDalImpl.java
View File

@ -118,7 +118,7 @@ public class RuleDalImpl implements RuleDal {
public PageResult<RuleHistoryVO> queryHistory(RuleHistoryQuery query) {
PageHelper.startPage(query.getPageNo(), query.getPageSize());
Example example = new Example(RuleHistoryVO.class);
Example example = new Example(RuleHistoryPO.class);
Example.Criteria criteria = example.createCriteria();
criteria.andEqualTo("ruleId", query.getRuleId());
List<RuleHistoryPO> list = ruleHistoryMapper.selectByExample(example);

11
radar-dal/src/main/java/com.pgmmers.radar/vo/model/DataListRecordVO.java
View File

@ -96,5 +96,14 @@ public class DataListRecordVO implements Serializable{
this.opt = opt;
}
@Override
public String toString() {
return "DataListRecordVO{" +
"id=" + id +
", dataListId=" + dataListId +
", dataRecord='" + dataRecord + '\'' +
", modelId=" + modelId +
", opt='" + opt + '\'' +
'}';
}
}

57
radar-dal/src/main/java/com.pgmmers.radar/vo/model/ModelConfParamVO.java Normal file
View File

@ -0,0 +1,57 @@
package com.pgmmers.radar.vo.model;
import java.io.Serializable;
/**
* <p>
* 机器学习模型配置，目前只考虑输入层为离散值的情况，不考虑需要词嵌入和融入卷积层，其中
* 离散值通过表达式取数从前置流程传递过来.
* </p>
*
* @author guor
* @date 2019/11/28
*/
public class ModelConfParamVO implements Serializable {
private Long id;
/**
* 参数的key
*/
private String feed;
/**
* 取数表达式，英文逗号分隔fields.deviceId，abstractions.log_uid_ip_1_day_qty
*/
private String expressions;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getFeed() {
return feed;
}
public void setFeed(String feed) {
this.feed = feed;
}
public String getExpressions() {
return expressions;
}
public void setExpressions(String expressions) {
this.expressions = expressions;
}
@Override
public String toString() {
return "ModelConfParamVO{" +
"id=" + id +
", feed='" + feed + '\'' +
", expressions='" + expressions + '\'' +
'}';
}
}

156
radar-dal/src/main/java/com.pgmmers.radar/vo/model/ModelConfVO.java Normal file
View File

@ -0,0 +1,156 @@
package com.pgmmers.radar.vo.model;
import java.io.Serializable;
import java.util.Date;
import java.util.List;
/**
* <p>
* 机器学习模型配置，定义模型文件路径和参数
* </p>
*
* @author guor
* @date 2019/11/28
*/
public class ModelConfVO implements Serializable {
/**
* 自增ID，主键
*/
private Long id;
private Long modelId;
/**
* 模型名称
*/
private String name;
/**
* 模型文件路径
*/
private String path;
/**
* tensorflow框架保存模型时设置的tag，非tensorflow模型此字段为空
*/
private String tag;
/**
* 参数列表
*/
private List<ModelConfParamVO> params;
/**
* 模型输出操作名称，predict_Y = tf.nn.softmax(softmax_before, name='predict')
*/
private String operation;
/**
* 模型更新时间
*/
private Date updateDate;
private String type;
private String comment;
private ModelConfParamVO confParam;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public List<ModelConfParamVO> getParams() {
return params;
}
public void setParams(List<ModelConfParamVO> params) {
this.params = params;
}
public String getPath() {
return path;
}
public void setPath(String path) {
this.path = path;
}
public Date getUpdateDate() {
return updateDate;
}
public void setUpdateDate(Date updateDate) {
this.updateDate = updateDate;
}
public String getTag() {
return tag;
}
public void setTag(String tag) {
this.tag = tag;
}
public String getOperation() {
return operation;
}
public void setOperation(String operation) {
this.operation = operation;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getComment() {
return comment;
}
public void setComment(String comment) {
this.comment = comment;
}
public Long getModelId() {
return modelId;
}
public void setModelId(Long modelId) {
this.modelId = modelId;
}
public ModelConfParamVO getConfParam() {
return confParam;
}
public void setConfParam(ModelConfParamVO confParam) {
this.confParam = confParam;
}
@Override
public String toString() {
return "ModelConfVO{" +
"id=" + id +
", modelId=" + modelId +
", name='" + name + '\'' +
", path='" + path + '\'' +
", tag='" + tag + '\'' +
", operation='" + operation + '\'' +
", updateDate=" + updateDate +
", type='" + type + '\'' +
", comment='" + comment + '\'' +
", confParam=" + confParam.toString() +
'}';
}
}

17
radar-dao/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>radar-dao</artifactId>
@ -28,16 +28,9 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.mybatis.generator</groupId>
<artifactId>mybatis-generator-core</artifactId>
<version>1.3.7</version>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
@ -46,12 +39,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
@ -83,8 +70,6 @@
</dependency>
</dependencies>
</plugin>
</plugins>
</build>

7
radar-dao/src/main/java/com/pgmmers/radar/mapper/ModelConfMapper.java Normal file
View File

@ -0,0 +1,7 @@
package com.pgmmers.radar.mapper;
import com.pgmmers.radar.model.ModelConfPO;
import tk.mybatis.mapper.common.Mapper;
public interface ModelConfMapper extends Mapper<ModelConfPO> {
}

7
radar-dao/src/main/java/com/pgmmers/radar/mapper/ModelConfParamMapper.java Normal file
View File

@ -0,0 +1,7 @@
package com.pgmmers.radar.mapper;
import com.pgmmers.radar.model.ModelConfParamPO;
import tk.mybatis.mapper.common.Mapper;
public interface ModelConfParamMapper extends Mapper<ModelConfParamPO> {
}

172
radar-dao/src/main/java/com/pgmmers/radar/model/ModelConfPO.java Normal file
View File

@ -0,0 +1,172 @@
package com.pgmmers.radar.model;
import java.util.Date;
import javax.persistence.*;
@Table(name = "engine_model_conf")
public class ModelConfPO {
@Id
@GeneratedValue(generator = "JDBC")
private Long id;
@Column(name = "model_id")
private Long modelId;
private String name;
private String path;
private String tag;
private String operation;
@Column(name = "update_date")
private Date updateDate;
private String type;
private String comment;
@Column(name = "create_time")
private Date createTime;
/**
* @return id
*/
public Long getId() {
return id;
}
/**
* @param id
*/
public void setId(Long id) {
this.id = id;
}
/**
* @return model_id
*/
public Long getModelId() {
return modelId;
}
/**
* @param modelId
*/
public void setModelId(Long modelId) {
this.modelId = modelId;
}
/**
* @return name
*/
public String getName() {
return name;
}
/**
* @param name
*/
public void setName(String name) {
this.name = name;
}
/**
* @return path
*/
public String getPath() {
return path;
}
/**
* @param path
*/
public void setPath(String path) {
this.path = path;
}
/**
* @return tag
*/
public String getTag() {
return tag;
}
/**
* @param tag
*/
public void setTag(String tag) {
this.tag = tag;
}
/**
* @return operation
*/
public String getOperation() {
return operation;
}
/**
* @param operation
*/
public void setOperation(String operation) {
this.operation = operation;
}
/**
* @return update_date
*/
public Date getUpdateDate() {
return updateDate;
}
/**
* @param updateDate
*/
public void setUpdateDate(Date updateDate) {
this.updateDate = updateDate;
}
/**
* @return type
*/
public String getType() {
return type;
}
/**
* @param type
*/
public void setType(String type) {
this.type = type;
}
/**
* @return comment
*/
public String getComment() {
return comment;
}
/**
* @param comment
*/
public void setComment(String comment) {
this.comment = comment;
}
/**
* @return create_time
*/
public Date getCreateTime() {
return createTime;
}
/**
* @param createTime
*/
public void setCreateTime(Date createTime) {
this.createTime = createTime;
}
}

73
radar-dao/src/main/java/com/pgmmers/radar/model/ModelConfParamPO.java Normal file
View File

@ -0,0 +1,73 @@
package com.pgmmers.radar.model;
import javax.persistence.*;
@Table(name = "engine_model_conf_param")
public class ModelConfParamPO {
@Id
@GeneratedValue(generator = "JDBC")
private Long id;
@Column(name = "mold_id")
private Long moldId;
private String feed;
private String expressions;
/**
* @return id
*/
public Long getId() {
return id;
}
/**
* @param id
*/
public void setId(Long id) {
this.id = id;
}
/**
* @return mold_id
*/
public Long getMoldId() {
return moldId;
}
/**
* @param moldId
*/
public void setMoldId(Long moldId) {
this.moldId = moldId;
}
/**
* @return feed
*/
public String getFeed() {
return feed;
}
/**
* @param feed
*/
public void setFeed(String feed) {
this.feed = feed;
}
/**
* @return expressions
*/
public String getExpressions() {
return expressions;
}
/**
* @param expressions
*/
public void setExpressions(String expressions) {
this.expressions = expressions;
}
}

8
radar-dao/src/main/resources/generator/generatorConfig.xml
View File

@ -86,6 +86,12 @@
<table tableName="DATA_MOBLE_INFO" domainObjectName="MobileInfo" selectByExampleQueryId="false">
<generatedKey column="ID" sqlStatement="JDBC" />
</table>
<table tableName="ENGINE_MODEL_CONF" domainObjectName="ModelConf" selectByExampleQueryId="false">
<generatedKey column="ID" sqlStatement="JDBC" />
</table>
<table tableName="ENGINE_MODEL_CONF_PARAM" domainObjectName="ModelConfParam" selectByExampleQueryId="false">
<generatedKey column="ID" sqlStatement="JDBC" />
</table>
</context>
</generatorConfiguration>

19
radar-dao/src/main/resources/mapping/ModelConfMapper.xml Normal file
View File

@ -0,0 +1,19 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.pgmmers.radar.mapper.ModelConfMapper">
<resultMap id="BaseResultMap" type="com.pgmmers.radar.model.ModelConfPO">
<!--
WARNING - @mbg.generated
-->
<id column="id" jdbcType="BIGINT" property="id" />
<result column="model_id" jdbcType="BIGINT" property="modelId" />
<result column="name" jdbcType="VARCHAR" property="name" />
<result column="path" jdbcType="VARCHAR" property="path" />
<result column="tag" jdbcType="VARCHAR" property="tag" />
<result column="operation" jdbcType="VARCHAR" property="operation" />
<result column="update_date" jdbcType="TIMESTAMP" property="updateDate" />
<result column="type" jdbcType="VARCHAR" property="type" />
<result column="comment" jdbcType="VARCHAR" property="comment" />
<result column="create_time" jdbcType="TIMESTAMP" property="createTime" />
</resultMap>
</mapper>

13
radar-dao/src/main/resources/mapping/ModelConfParamMapper.xml Normal file
View File

@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.pgmmers.radar.mapper.ModelConfParamMapper">
<resultMap id="BaseResultMap" type="com.pgmmers.radar.model.ModelConfParamPO">
<!--
WARNING - @mbg.generated
-->
<id column="id" jdbcType="BIGINT" property="id" />
<result column="mold_id" jdbcType="BIGINT" property="moldId" />
<result column="feed" jdbcType="VARCHAR" property="feed" />
<result column="expressions" jdbcType="VARCHAR" property="expressions" />
</resultMap>
</mapper>

45
radar-engine/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -27,11 +27,6 @@
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
@ -42,12 +37,6 @@
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
<artifactId>radar-service</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
<artifactId>radar-service-impl</artifactId>
@ -64,45 +53,25 @@
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
</dependency>
</dependencies>
<build>
@ -111,14 +80,6 @@
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.1.7.RELEASE</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

6
radar-engine/src/main/java/com/pgmmers/radar/EngineApplication.java
View File

@ -2,13 +2,17 @@ package com.pgmmers.radar;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.data.mongo.MongoDataAutoConfiguration;
import org.springframework.boot.autoconfigure.elasticsearch.rest.RestClientAutoConfiguration;
import org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration;
import tk.mybatis.spring.annotation.MapperScan;
@SpringBootApplication
@SpringBootApplication(exclude = {MongoAutoConfiguration.class, MongoDataAutoConfiguration.class, RestClientAutoConfiguration.class})
@MapperScan("com.pgmmers.radar.mapper")
public class EngineApplication {
public static void main(String[] args) {
System.setProperty("es.set.netty.runtime.available.processors", "false");
SpringApplication.run(EngineApplication.class, args);
}

47
radar-engine/src/main/java/com/pgmmers/radar/config/EsConfig.java Normal file
View File

@ -0,0 +1,47 @@
package com.pgmmers.radar.config;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.net.InetAddress;
@Configuration
public class EsConfig {
private static final Logger logger = LoggerFactory.getLogger(EsConfig.class);
@Value("${elasticsearch.ip}")
private String hostName;
@Value("${elasticsearch.port}")
private String port;
@Value("${elasticsearch.cluster.name}")
private String clusterName;
@Value("${elasticsearch.pool-size}")
private String poolSize;
@Bean(name = "transportClient")
public TransportClient transportClient() {
logger.info("Elasticsearch初始化开始。。。。。");
TransportClient transportClient = null;
try {
// 配置信息
Settings esSetting = Settings.builder()
.put("cluster.name", clusterName) //集群名字
.put("client.transport.sniff", true)//增加嗅探机制，找到ES集群
.put("thread_pool.search.size", Integer.parseInt(poolSize))//增加线程池个数，暂时设为5
.build();
//配置信息Settings自定义
transportClient = new PreBuiltTransportClient(esSetting);
TransportAddress transportAddress = new TransportAddress(InetAddress.getByName(hostName), Integer.valueOf(port));
transportClient.addTransportAddresses(transportAddress);
} catch (Exception e) {
logger.error("elasticsearch TransportClient create error!!", e);
}
return transportClient;
}
}

2
radar-engine/src/main/java/com/pgmmers/radar/config/SwaggerConfig.java
View File

@ -21,7 +21,7 @@ public class SwaggerConfig {
.apiInfo(buildApiInf())
.select()
.apis(RequestHandlerSelectors.basePackage("com.pgmmers.radar.controller"))
.paths(PathSelectors.any())
.paths(PathSelectors.regex("/services/.*"))
.build();
}

22
radar-engine/src/main/java/com/pgmmers/radar/controller/IndexController.java Normal file
View File

@ -0,0 +1,22 @@
package com.pgmmers.radar.controller;
import com.pgmmers.radar.service.common.CommonResult;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import javax.servlet.http.HttpServletRequest;
/**
* engine 启动首页.
* @author wangfeihu
*/
@RestController
public class IndexController {
@GetMapping(value = {"/", ""})
public CommonResult index(HttpServletRequest request) {
CommonResult result = new CommonResult(Boolean.TRUE, "100", "Engine is running");
result.getData().put("swagger url:", request.getRequestURL() + "swagger-ui.html");
return result;
}
}

19
radar-engine/src/main/resources/application.yml
View File

@ -47,12 +47,16 @@ mongodb:
url: mongodb://localhost:27017/radar
mobile:
info:
path: D:/soft/moble_info.csv
path: D:/radar/moble_info.csv # 手机号码归属地文件
ip2region:
db:
path: D:/soft/ip2region.db
elksearch:
server: localhost
path: D:/radar/ip2region.db # IP地址库文件
elasticsearch:
ip: localhost
port: 9300
pool-size: 5
cluster:
name: elasticsearch
url: http://localhost:9200
logging:
level:
@ -61,8 +65,9 @@ logging:
com.pgmmers.radar.mapper: debug
sys:
conf:
app: engine
entity-duplicate-insert: false
mongo-restore-days: 93
app: engine # admin 或者 engine， 根据启动的项目名称进行选择
entity-duplicate-insert: false # 事件是否允许重复插入
mongo-restore-days: 93 # 事件保存时间，默认3个月
workdir: d:\\radar # 工作目录
server:
port: 9090

30
radar-service-impl/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -13,29 +13,14 @@
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.9</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
@ -57,14 +42,19 @@
<artifactId>ip2region</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.tensorflow</groupId>
<artifactId>tensorflow</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
</dependency>
</dependencies>
</project>

22
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/core/RiskAnalysisEngineServiceImpl.java
View File

@ -17,7 +17,9 @@ import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.http.*;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import java.util.Date;
import java.util.HashMap;
@ -47,6 +49,12 @@ public class RiskAnalysisEngineServiceImpl implements RiskAnalysisEngineService
@Autowired
private ValidateService validateService;
@Autowired
private RestTemplate restTemplate;
@Value("${elasticsearch.url}")
private String elasticsearchUrl;
@Override
public CommonResult uploadInfo(String modelGuid, String reqId, String jsonInfo) {
logger.info("req info:{},{},{}", modelGuid, reqId, jsonInfo);
@ -132,7 +140,19 @@ public class RiskAnalysisEngineServiceImpl implements RiskAnalysisEngineService
* @param info event info and analyze result.
*/
private void sendResult(String modelGuid, String reqId, String info) {
//TODO send to MQ.
// 这里可以根据情况进行异步处理。
send2ES(modelGuid, info);
}
private void send2ES(String guid, String json) {
String url = elasticsearchUrl + "/" + guid.toLowerCase() + "/" + "radar";
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<String> requestEntity = new HttpEntity<>(json, headers);
ResponseEntity<String> result = restTemplate.postForEntity(url, requestEntity, String.class, new Object[]{});
logger.info("es result:{}", result);
}
}

39
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/dnn/EstimatorContainer.java Normal file
View File

@ -0,0 +1,39 @@
package com.pgmmers.radar.service.impl.dnn;
import com.pgmmers.radar.service.dnn.Estimator;
import com.pgmmers.radar.service.model.ModelConfService;
import com.pgmmers.radar.vo.model.ModelConfVO;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import javax.annotation.Resource;
import java.util.HashMap;
import java.util.Map;
@Component
public class EstimatorContainer {
private Map<String, Estimator> estimatorMap = new HashMap<>();
@Resource
private ModelConfService modelConfService;
@Autowired
public void set(Estimator[] estimators) {
for (Estimator estimator : estimators) {
estimatorMap.put(estimator.getType(), estimator);
}
}
public Estimator getByType(String type) {
return estimatorMap.get(type);
}
public Estimator getByModelId(Long modelId) {
ModelConfVO mold = modelConfService.getByModelId(modelId);
if (mold == null) {
return null;
}
return getByType(mold.getType());
}
}

123
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/dnn/TensorDnnEstimator.java Normal file
View File

@ -0,0 +1,123 @@
package com.pgmmers.radar.service.impl.dnn;
import com.pgmmers.radar.service.dnn.Estimator;
import com.pgmmers.radar.service.model.ModelConfService;
import com.pgmmers.radar.vo.model.ModelConfParamVO;
import com.pgmmers.radar.vo.model.ModelConfVO;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.tensorflow.SavedModelBundle;
import org.tensorflow.Session;
import org.tensorflow.Tensor;
import javax.annotation.Resource;
import java.io.File;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@Component
public class TensorDnnEstimator implements Estimator {
private static final Logger LOGGER = LoggerFactory.getLogger(TensorDnnEstimator.class);
@Resource
private ModelConfService modelConfService;
private Map<Long, SavedModelBundle> modelBundleMap = new HashMap<>();
@Value("${sys.conf.workdir}")
private String workDir;
@Override
public float predict(Long modelId, Map<String, Map<String, ?>> data) {
ModelConfVO mold = modelConfService.getByModelId(modelId);
if (mold == null) {
LOGGER.debug("没有找到模型配置，ModelId：{}", modelId);
return 0;
}
SavedModelBundle modelBundle = loadAndCacheModel(mold);
if (modelBundle == null) {
LOGGER.warn("模型文件不存在或加载失败，ModelId：{}", modelId);
return 0;
}
Session tfSession = modelBundle.session();
try {
List<ModelConfParamVO> params = mold.getParams();
Session.Runner runner = tfSession.runner();
for (ModelConfParamVO moldParam : params) {
runner.feed(moldParam.getFeed(), convert2Tensor(moldParam, data));
}
Tensor<?> output = runner.fetch(mold.getOperation()).run().get(0);
float[][] results = new float[1][1];
output.copyTo(results);
return results[0][0];
} catch (Exception e) {
LOGGER.error("模型调用失败，ModelId:" + modelId, e);
}
return 0;
}
private Tensor<?> convert2Tensor(ModelConfParamVO moldParam, Map<String, Map<String, ?>> data) {
String expressions = moldParam.getExpressions();
if (StringUtils.isEmpty(expressions)) {
return Tensor.create(new float[1][1]);
}
String[] expList = expressions.split(",");
float[][] vec = new float[1][expList.length];
int a = 0;
for (String s : expList) {
float xn = 0;
String[] ss = s.split("\\.");//fields.deviceId，abstractions.log_uid_ip_1_day_qty
Map<String, ?> stringMap = data.get(ss[0]);
if (stringMap != null) {
xn = Float.parseFloat(String.valueOf(stringMap.get(ss[1])));
}
vec[0][a++] = xn;
}
return Tensor.create(vec);
}
private synchronized SavedModelBundle loadAndCacheModel(ModelConfVO mold) {
SavedModelBundle modelBundle = modelBundleMap.get(mold.getId());
if (modelBundle == null) {
String path = workDir + "\\" + mold.getPath();
String decomposePath = path.substring(0, path.lastIndexOf("."));
File file = new File(decomposePath);
if (file.exists() && file.isDirectory()) {
// 模型加载，比较耗时
try {
modelBundle = SavedModelBundle.load(mold.getPath(), mold.getTag());
modelBundleMap.put(mold.getId(), modelBundle);
} catch (Exception e) {
LOGGER.warn("模型加载失败，MoldId：{}", mold.getId());
}
}
}
return modelBundle;
}
@Override
public String getType() {
return Estimator.TYPE_TENSOR_DNN;
}
public static void main(String[] args) {
SavedModelBundle modelBundle = SavedModelBundle.load("d:/radar01", "serve");
Session tfSession = modelBundle.session();
try {
Session.Runner runner = tfSession.runner();
float[][] aa = new float[1][6];
aa[0] = new float[]{20f, 1f, 1f, 1f, 10f, 2f};
runner.feed("input_x", Tensor.create(aa));
Tensor<?> output = runner.fetch("output_y/BiasAdd").run().get(0);
float[][] results = new float[1][1];
output.copyTo(results);
System.out.println(results[0][0]);
} catch (Exception e) {
e.printStackTrace();
} finally {
tfSession.close();
}
}
}

13
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/engine/AntiFraudEngineImpl.java
View File

@ -5,9 +5,11 @@ import com.pgmmers.radar.enums.AggregateType;
import com.pgmmers.radar.enums.FieldType;
import com.pgmmers.radar.enums.Operator;
import com.pgmmers.radar.enums.StatusType;
import com.pgmmers.radar.service.dnn.Estimator;
import com.pgmmers.radar.service.engine.AggregateCommand;
import com.pgmmers.radar.service.engine.AntiFraudEngine;
import com.pgmmers.radar.service.engine.vo.*;
import com.pgmmers.radar.service.impl.dnn.EstimatorContainer;
import com.pgmmers.radar.service.model.*;
import com.pgmmers.radar.util.DateUtils;
import com.pgmmers.radar.util.GroovyScriptUtil;
@ -51,6 +53,9 @@ public class AntiFraudEngineImpl implements AntiFraudEngine {
@Autowired
private RuleService ruleService;
@Autowired
private EstimatorContainer estimatorContainer;
@Override
public AbstractionResult executeAbstraction(Long modelId, Map<String, Map<String, ?>> data) {
AbstractionResult result = new AbstractionResult();
@ -243,9 +248,13 @@ public class AntiFraudEngineImpl implements AntiFraudEngine {
@Override
public AdaptationResult executeAdaptation(Long modelId, Map<String, Map<String, ?>> data) {
AdaptationResult result = new AdaptationResult();
// TODO Auto-generated method stub
Estimator estimator = estimatorContainer.getByModelId(modelId);
if(estimator != null) {
float score = estimator.predict(modelId, data);
result.getAdaptationMap().put("score", score);
}
result.setSuccess(true);
data.put("adapations", new HashMap<String, Object>());
data.put("adaptations", result.getAdaptationMap());
return result;
}

31
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/logs/EventServiceImpl.java
View File

@ -1,5 +1,6 @@
package com.pgmmers.radar.service.impl.logs;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.pgmmers.radar.dal.bean.EventQuery;
@ -69,7 +70,8 @@ public class EventServiceImpl implements EventService {
query.getPageSize());
SearchHit[] hits = hitsRet.getHits();
for (SearchHit hit : hits) {
String info = hit.sourceRef().toUtf8();
String info = hit.getSourceRef().utf8ToString();
list.add(JSONObject.parse(info));
list.add(info);
}
return list;
@ -112,8 +114,9 @@ public class EventServiceImpl implements EventService {
hits = hitsRet.getHits();
for (SearchHit hit : hits) {
String info = hit.sourceRef().toUtf8();
list.add(info);
//String info = hit.sourceRef().toUtf8();
String info = hit.getSourceRef().utf8ToString();
list.add(JSONObject.parse(info));
}
pageResult = new PageResult<>(term.getPageNo(),
term.getPageSize(), (int) hitsRet.getTotalHits(), list);
@ -161,7 +164,7 @@ public class EventServiceImpl implements EventService {
// 创建第一种字体样式（用于列名）
f.setFontHeightInPoints((short) 10);
f.setColor(IndexedColors.BLACK.getIndex());
f.setBoldweight(Font.BOLDWEIGHT_BOLD);
// f.setBoldweight(Font.BOLDWEIGHT_BOLD);
// 创建第二种字体样式（用于值）
f2.setFontHeightInPoints((short) 10);
@ -173,19 +176,19 @@ public class EventServiceImpl implements EventService {
// 设置第一种单元格的样式（用于列名）
cs.setFont(f);
cs.setBorderLeft(CellStyle.BORDER_THIN);
cs.setBorderRight(CellStyle.BORDER_THIN);
cs.setBorderTop(CellStyle.BORDER_THIN);
cs.setBorderBottom(CellStyle.BORDER_THIN);
cs.setAlignment(CellStyle.ALIGN_CENTER);
// cs.setBorderLeft(CellStyle.BORDER_THIN);
// cs.setBorderRight(CellStyle.BORDER_THIN);
// cs.setBorderTop(CellStyle.BORDER_THIN);
// cs.setBorderBottom(CellStyle.BORDER_THIN);
// cs.setAlignment(CellStyle.ALIGN_CENTER);
// 设置第二种单元格的样式（用于值）
cs2.setFont(f2);
cs2.setBorderLeft(CellStyle.BORDER_THIN);
cs2.setBorderRight(CellStyle.BORDER_THIN);
cs2.setBorderTop(CellStyle.BORDER_THIN);
cs2.setBorderBottom(CellStyle.BORDER_THIN);
cs2.setAlignment(CellStyle.ALIGN_CENTER);
// cs2.setBorderLeft(CellStyle.BORDER_THIN);
// cs2.setBorderRight(CellStyle.BORDER_THIN);
// cs2.setBorderTop(CellStyle.BORDER_THIN);
// cs2.setBorderBottom(CellStyle.BORDER_THIN);
// cs2.setAlignment(CellStyle.ALIGN_CENTER);
// 设置列名
for (int i = 0; i < titleList4Field.size(); i++) {
Cell cell = row.createCell(i);

10
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/model/AbstractionServiceImpl.java
View File

@ -9,6 +9,7 @@ import com.pgmmers.radar.service.cache.CacheService;
import com.pgmmers.radar.service.cache.SubscribeHandle;
import com.pgmmers.radar.service.common.CommonResult;
import com.pgmmers.radar.service.model.AbstractionService;
import com.pgmmers.radar.util.GroovyScriptUtil;
import com.pgmmers.radar.vo.model.AbstractionVO;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
@ -79,10 +80,17 @@ public class AbstractionServiceImpl implements AbstractionService, SubscribeHand
@Override
public CommonResult save(AbstractionVO abstraction) {
CommonResult result = new CommonResult();
if (abstraction.getId() != null) {
AbstractionVO oldAbs = abstractionDal.get(abstraction.getId());
// 如果规则有更新，以前的编译好的groovy 对象用不到了，应该删除。
if (!oldAbs.getRuleScript().equals(abstraction.getRuleScript())) {
GroovyScriptUtil.removeInactiveScript(oldAbs.getRuleScript());
}
}
int count = abstractionDal.save(abstraction);
if (count > 0) {
if(StringUtils.isEmpty(abstraction.getName())){
abstraction.setName("abstraction_"+abstraction.getId());
abstraction.setName("abstraction_" + abstraction.getId());
abstractionDal.save(abstraction);
}

1
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/model/DataListsServiceImpl.java
View File

@ -306,6 +306,7 @@ public class DataListsServiceImpl implements DataListsService, SubscribeHandle {
public CommonResult batchImportDataRecord(List<DataListRecordVO> list, Long dataListId) {
CommonResult result = new CommonResult();
for (DataListRecordVO dataListRecord : list) {
dataListRecord.setDataListId(dataListId);
int count = dataListDal.saveRecord(dataListRecord);
if (count > 0) {
// 通知更新

27
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/model/ModelConfParamServiceImpl.java Normal file
View File

@ -0,0 +1,27 @@
package com.pgmmers.radar.service.impl.model;
import com.pgmmers.radar.dal.model.ModelConfDal;
import com.pgmmers.radar.service.model.ModelConfParamService;
import com.pgmmers.radar.vo.model.ModelConfParamVO;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class ModelConfParamServiceImpl implements ModelConfParamService {
@Autowired
private ModelConfDal modelConfDal;
@Override
public ModelConfParamVO get(Long id) {
ModelConfParamVO paramVO;
paramVO = modelConfDal.getParamById(id);
return paramVO;
}
@Override
public ModelConfParamVO save(ModelConfParamVO modelConfParam) {
return modelConfDal.saveParam(modelConfParam);
}
}

29
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/model/ModelConfServiceImpl.java Normal file
View File

@ -0,0 +1,29 @@
package com.pgmmers.radar.service.impl.model;
import com.pgmmers.radar.dal.model.ModelConfDal;
import com.pgmmers.radar.service.model.ModelConfService;
import com.pgmmers.radar.vo.model.ModelConfVO;
import org.springframework.stereotype.Service;
import javax.annotation.Resource;
@Service
public class ModelConfServiceImpl implements ModelConfService {
@Resource
private ModelConfDal modelConfDal;
@Override
public ModelConfVO get(Long id) {
return modelConfDal.get(id);
}
@Override
public ModelConfVO getByModelId(Long modelId) {
return modelConfDal.getByModelId(modelId);
}
@Override
public ModelConfVO save(ModelConfVO modelConf) {
return modelConfDal.save(modelConf);
}
}

16
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/model/ModelServiceImpl.java
View File

@ -240,7 +240,7 @@ public class ModelServiceImpl implements ModelService, SubscribeHandle {
// field mapping
JSONObject fieldJson = new JSONObject();
String base = "{\"type\": \"%s\",\"index\": \"not_analyzed\"}";
String base = "{\"type\": \"%s\"}";
for (FieldVO field : fields) {
String fieldType = field.getFieldType();
String elaType = convertFieldType2ElasticType(fieldType);
@ -256,9 +256,13 @@ public class ModelServiceImpl implements ModelService, SubscribeHandle {
String columns = plugin.getMeta();
if (columns == null) {
String fieldType = plugin.getType();
String elaType = convertFieldType2ElasticType(fieldType);
String tmp = String.format(base, elaType);
preItemJson.put(item.getDestField(), JSON.parseObject(tmp));
if(fieldType.equals("JSON")) {
//TODO: json类型需要另外处理
} else {
String elaType = convertFieldType2ElasticType(fieldType);
String tmp = String.format(base, elaType);
preItemJson.put(item.getDestField(), JSON.parseObject(tmp));
}
} else {
String meta = plugin.getMeta();
List<JSONObject> fieldsJson = JSON.parseArray(meta, JSONObject.class);
@ -334,7 +338,7 @@ public class ModelServiceImpl implements ModelService, SubscribeHandle {
String tmp;
switch (type) {
case STRING:
tmp = "string";
tmp = "keyword";
break;
case INTEGER:
tmp = "integer";
@ -346,7 +350,7 @@ public class ModelServiceImpl implements ModelService, SubscribeHandle {
tmp = "double";
break;
default:
tmp = "string";
tmp = "text";
break;
}
return tmp;

28
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/model/RuleServiceImpl.java
View File

@ -14,8 +14,10 @@ import com.pgmmers.radar.service.cache.SubscribeHandle;
import com.pgmmers.radar.service.common.CommonResult;
import com.pgmmers.radar.service.model.RuleService;
import com.pgmmers.radar.service.search.SearchEngineService;
import com.pgmmers.radar.util.GroovyScriptUtil;
import com.pgmmers.radar.vo.model.*;
import org.apache.commons.lang3.StringUtils;
import org.elasticsearch.index.query.QueryBuilders;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
@ -43,7 +45,7 @@ public class RuleServiceImpl implements RuleService, SubscribeHandle {
@Autowired
private SearchEngineService searchService;
public Map<Long, List<RuleVO>> contextMap = new HashMap<Long, List<RuleVO>>();
public Map<Long, List<RuleVO>> contextMap = new HashMap<>();
public List<RuleVO> listRule(Long modelId, Long activationId, Integer status) {
return modelDal.listRules(modelId, activationId, status);
@ -89,6 +91,13 @@ public class RuleServiceImpl implements RuleService, SubscribeHandle {
@Override
public CommonResult save(RuleVO rule,String merchantCode) {
CommonResult result = new CommonResult();
if (rule.getId() != null) {
RuleVO oldRule = ruleDal.get(rule.getId());
// 如果规则有更新，以前的编译好的groovy 对象用不到了，应该删除。
if (!oldRule.getScripts().equals(rule.getScripts())) {
GroovyScriptUtil.removeInactiveScript(oldRule.getScripts());
}
}
int count = ruleDal.save(rule);
if (count > 0) {
if(StringUtils.isEmpty(rule.getName())){
@ -139,9 +148,9 @@ public class RuleServiceImpl implements RuleService, SubscribeHandle {
@Override
public CommonResult getHitSorts(Long modelId) {
CommonResult result = new CommonResult();
Set<RuleHitsVO> treeSet = new TreeSet<RuleHitsVO>();
Set<RuleHitsVO> treeSet = new TreeSet<>();
ModelVO model = modelDal.getModelById(modelId);
String elaQuery = "{\"query\":{\"term\":{\"hitsDetail.${activationName}.rule_${ruleId}.key\":\"${ruleId}\"}}}";
String keyTempl = "hitsDetail.${activationName}.rule_${ruleId}.key";
ActivationQuery actQuery = new ActivationQuery();
actQuery.setModelId(modelId);
PageResult<ActivationVO> actResult = activationDal.query(actQuery);
@ -153,20 +162,17 @@ public class RuleServiceImpl implements RuleService, SubscribeHandle {
PageResult<RuleVO> page = ruleDal.query(query);
List<RuleVO> list = page.getList();
for (RuleVO rule : list) {
String tmpQuery = elaQuery.replace("${activationName}",
act.getActivationName());
tmpQuery = tmpQuery.replace("${ruleId}", rule.getId() + "");
String keyStr = keyTempl.replace("${activationName}", act.getActivationName());
keyStr = keyStr.replace("${ruleId}", rule.getId() + "");
long qty = 0;
try {
qty = searchService.count(model.getGuid().toLowerCase(),
"radar", tmpQuery, null);
"radar", QueryBuilders.termQuery(keyStr,rule.getId() + ""), null);
} catch (Exception e) {
logger.error("search error", e);
qty = 0;
}
if (qty <= 0) {
continue;
} else {
if (qty > 0){
RuleHitsVO hit = new RuleHitsVO();
hit.setId(rule.getId());
hit.setActivationName(act.getActivationName());

133
radar-service-impl/src/main/java/com/pgmmers/radar/service/impl/search/SearchEngineServiceImpl.java
View File

@ -1,33 +1,131 @@
package com.pgmmers.radar.service.impl.search;
import com.pgmmers.radar.service.search.SearchEngineService;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequestBuilder;
import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchType;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.client.IndicesAdminClient;
import org.elasticsearch.client.Requests;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.aggregations.bucket.terms.Terms;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import java.util.Map;
@Service
public class SearchEngineServiceImpl implements SearchEngineService {
public static Logger logger = LoggerFactory
.getLogger(SearchEngineServiceImpl.class);
@Autowired
private TransportClient client;
@Override
public SearchHits search(String index, String type, String queryJson, String filterJson, Integer offset, Integer limit) {
return null;
SearchResponse response = null;
SearchRequestBuilder builder = client.prepareSearch(index)
.setTypes(type).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(offset).setSize(limit).setExplain(true);
if (!StringUtils.isEmpty(queryJson)) {
builder.setQuery(QueryBuilders.queryStringQuery(queryJson));
}
if (!StringUtils.isEmpty(filterJson)) {
builder.setPostFilter(QueryBuilders.queryStringQuery(filterJson));
}
response = builder.execute().actionGet();
RestStatus status = response.status();
if (status.equals(RestStatus.OK)) {
SearchHits hits = response.getHits();
return hits;
} else {
return null;
}
}
@Override
public SearchHits search(String index, String type, Map<String, Object> queryMap, Map<String, Object> filterMap, Integer offset, Integer limit) {
return null;
QueryBuilder query = null;
QueryBuilder filter = null;
if (queryMap != null) {
Object entityId = queryMap.get("entityId");
if (!StringUtils.isEmpty(entityId)) {
query = QueryBuilders.termQuery(
"fields." + queryMap.get("entityName").toString(),
entityId.toString());
}
}
if (filterMap != null) {
Long beginTime = (Long) filterMap.get("beginTime");
Long endTime = (Long) filterMap.get("endTime");
filter = QueryBuilders
.rangeQuery("fields." + filterMap.get("refDate").toString())
.from(beginTime.longValue()).to(endTime.longValue());
}
return search(index, type, query, filter, offset, limit);
}
@Override
public SearchHits search(String index, String type, QueryBuilder query, QueryBuilder filter, Integer offset, Integer limit) {
return null;
SearchResponse response = null;
SearchRequestBuilder builder = client.prepareSearch(index)
.setTypes(type).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(offset).setSize(limit).setExplain(true);
if (query != null) {
builder.setQuery(query);
}
if (filter != null) {
builder.setPostFilter(filter);
}
response = builder.execute().actionGet();
RestStatus status = response.status();
if (status.equals(RestStatus.OK)) {
SearchHits hits = response.getHits();
return hits;
} else {
return null;
}
}
@Override
public Long count(String index, String type, String query, String filter) {
return null;
return 0L;
}
@Override
public Long count(String index, String type, QueryBuilder query, QueryBuilder filter) {
SearchResponse response = null;
SearchRequestBuilder builder = client.prepareSearch(index)
.setTypes(type).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setSize(0).setExplain(true);
if (!StringUtils.isEmpty(query)) {
builder.setQuery(query);
}
if (!StringUtils.isEmpty(filter)) {
builder.setPostFilter(filter);
}
response = builder.execute().actionGet();
RestStatus status = response.status();
if (status.equals(RestStatus.OK)) {
SearchHits hits = response.getHits();
return hits.getTotalHits();
} else {
return null;
}
}
@Override
@ -47,7 +145,30 @@ public class SearchEngineServiceImpl implements SearchEngineService {
@Override
public boolean createIndex(String index, String indexAlias, String type, String jsonMapping) {
IndicesAdminClient adminClient = client.admin().indices();
DeleteIndexRequest request = Requests.deleteIndexRequest(index);
adminClient.delete(request);
adminClient.prepareCreate(index).get();
AcknowledgedResponse resp;
PutMappingRequestBuilder builder = adminClient.preparePutMapping(index);
if (!StringUtils.isEmpty(jsonMapping)) {
builder.setType(type).setSource(jsonMapping, XContentType.JSON);
} else {
return false;
}
resp = builder.get();
if (resp.isAcknowledged()) {
// 创建别名
AcknowledgedResponse aliasResp = adminClient.prepareAliases().addAlias(index, indexAlias).get();
if (aliasResp.isAcknowledged()) {
return true;
} else {
return false;
}
return true;
} else {
return false;
}
}
}

10
radar-service/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>radar</artifactId>
<groupId>com.pgmmers</groupId>
<version>1.0.2-SNAPSHOT</version>
<version>1.0.3-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -19,10 +19,6 @@
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</dependency>
<dependency>
<groupId>com.pgmmers</groupId>
<artifactId>radar-dal</artifactId>
@ -35,8 +31,8 @@
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
</dependency>

11
radar-service/src/main/java/com/pgmmers/radar/service/common/CommonResult.java
View File

@ -58,6 +58,17 @@ public class CommonResult implements Serializable{
this.data = data;
}
public CommonResult () {
}
public CommonResult(boolean success, String code, String msg) {
this.success = success;
this.code = code;
this.msg = msg;
}
@Override
public String toString() {
return "CommonResult [success=" + success + ", msg=" + msg + ", code=" + code + "]";

33
radar-service/src/main/java/com/pgmmers/radar/service/dnn/Estimator.java Normal file
View File

@ -0,0 +1,33 @@
package com.pgmmers.radar.service.dnn;
import java.util.Map;
/**
* <p>
* 机器学习模型执行器接口
* </p>
* <p>
* 该接口内置TensorFlow实现，目前版本只考虑输入层为离散值的情况，不考虑词嵌入和融入卷积层，其中
* 离散值通过表达式取数从前置流程传递过来，模型的预测结果为事件评分。
* </p>
* <p>
* 模型抽象：y=f(x)
* </p>
*
* @author guor
* @date 2019/11/28
*/
public interface Estimator {
/**
* 线性回归模型
*/
String TYPE_REGRESSION = "REGRESSION";
/**
* 基于TensorFlow实现的神经网络模型
*/
String TYPE_TENSOR_DNN = "TENSOR_DNN";
float predict(Long modelId, Map<String, Map<String, ?>> data);
String getType();
}

11
radar-service/src/main/java/com/pgmmers/radar/service/model/ModelConfParamService.java Normal file
View File

@ -0,0 +1,11 @@
package com.pgmmers.radar.service.model;
import com.pgmmers.radar.vo.model.ModelConfParamVO;
public interface ModelConfParamService {
ModelConfParamVO get(Long id);
ModelConfParamVO save(ModelConfParamVO modelConfParam);
}

12
radar-service/src/main/java/com/pgmmers/radar/service/model/ModelConfService.java Normal file
View File

@ -0,0 +1,12 @@
package com.pgmmers.radar.service.model;
import com.pgmmers.radar.vo.model.ModelConfVO;
public interface ModelConfService {
ModelConfVO get(Long id);
ModelConfVO getByModelId(Long modelId);
ModelConfVO save(ModelConfVO modelConf);
}

4
radar-service/src/main/java/com/pgmmers/radar/service/search/SearchEngineService.java
View File

@ -21,7 +21,9 @@ public interface SearchEngineService {
SearchHits search(String index, String type, QueryBuilder query, QueryBuilder filter, Integer offset, Integer limit);
Long count(String index, String type, String query, String filter);
Long count(String index, String type, QueryBuilder query, QueryBuilder filter);
double avg(String index, String type, String field, String filter);
double max(String index, String type, String field, String filter);

BIN
resources/radar-tran-v1.zip Normal file
View File

Binary file not shown.

48
sql/radar-1.0.3.sql Normal file
View File

@ -0,0 +1,48 @@
/*
Navicat MySQL Data Transfer
Source Server : test@172.30.0.6
Source Server Version : 50726
Source Host : 172.30.0.6:3306
Source Database : radar
Target Server Type : MYSQL
Target Server Version : 50726
File Encoding : 65001
Date: 2019-12-24 18:02:12
*/
SET FOREIGN_KEY_CHECKS=0;
-- ----------------------------
-- Table structure for engine_model_conf
-- ----------------------------
DROP TABLE IF EXISTS `engine_model_conf`;
CREATE TABLE `engine_model_conf` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`model_id` bigint(20) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`path` varchar(255) DEFAULT NULL,
`tag` varchar(255) DEFAULT NULL,
`operation` varchar(255) DEFAULT NULL,
`update_date` datetime DEFAULT NULL,
`type` varchar(255) DEFAULT NULL,
`comment` varchar(128) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
-- ----------------------------
-- Table structure for engine_model_conf_param
-- ----------------------------
DROP TABLE IF EXISTS `engine_model_conf_param`;
CREATE TABLE `engine_model_conf_param` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`mold_id` bigint(20) DEFAULT NULL,
`feed` varchar(255) DEFAULT NULL,
`expressions` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;

4
webapp/src/component/Index.jsx
View File

@ -33,6 +33,10 @@ export default class Index extends React.Component{
<div className="ant-layout-logo">
<Tooltip title="统计报表"><Link to="/report"><Icon type="line-chart" /></Link></Tooltip>
</div>
<div className="ant-layout-logo">
<Tooltip title="配置中心"><Link to="/config"><Icon type="setting" /></Link></Tooltip>
</div>
<div className="ant-layout-logo" style={{float:"right",marginRight:0}}>
<Tooltip title="退出登录"><a onClick={this.handleLogout}><Icon type="logout" /></a></Tooltip>
</div>

2
webapp/src/component/activation/HistoryRecordList.jsx
View File

@ -49,7 +49,7 @@ export default class HistoryRecordList extends Component{
param.ruleId=this.props.params.ruleId;
//此处为单条策略历史记录接口的获取，参数中应该有本条策略的id，否则无法区分
FetchUtil('/ruleHistory','POST',JSON.stringify(param),
FetchUtil('/rule/ruleHistory','POST',JSON.stringify(param),
(data) => {
this.setState({loading:false});
this.setState({

50
webapp/src/component/config/ConfigCenter.jsx Normal file
View File

@ -0,0 +1,50 @@
import React from 'react';
import {Breadcrumb,Menu,Icon,Form,Select, Card, Row, Col} from 'antd';
import {Link} from 'react-router';
const FormItem=Form.Item;
const Option = Select.Option;
import {FetchUtil} from '../utils/fetchUtil';
export default class ConfigCenter extends React.Component{
render() {
return (
<div className="ant-layout-wrapper">
<div className="ant-layout-breadcrumb">
<Breadcrumb>
<Breadcrumb.Item>首页</Breadcrumb.Item>
<Breadcrumb.Item>配置中心</Breadcrumb.Item>
</Breadcrumb>
</div>
<div className="ant-layout-container">
<Row gutter={8}>
<Col span={6}>
<Card size="small" title="数据列表配置" extra={<a >More</a>} style={{ width: 300 }}>
<p>全局黑白名单数据的配置。。。</p>
<p>&nbsp;</p>
<Link >进入配置</Link>
</Card>
</Col>
<Col span={6}>
<Card title="待开发" extra={<a >More</a>} style={{ width: 300 }}>
<p>待开发。。。</p>
<p>&nbsp;</p>
<Link >进入配置</Link>
</Card>
</Col>
<Col span={6} />
<Col span={6} />
</Row>
<Row>
</Row>
</div>
</div>
);
}
}

280
webapp/src/component/config/LearningConfig.jsx Normal file
View File

@ -0,0 +1,280 @@
import React from 'react';
import {Breadcrumb, Menu, Icon, Form, Upload, Input, Select, Card, Row, Col,Button, Tooltip, Tag} from 'antd';
const FormItem=Form.Item;
const Option = Select.Option;
import {FetchUtil} from '../utils/fetchUtil';
export default class LearningConfig extends React.Component{
constructor(props){
super(props);
this.state={
visible:false,
destField:'',
label:'',
sourceField:'',
sourceLabel:'',
plugin:'Tensorflow',
status:1,
args:'',
reqType:'GET',
configJson:'',
tags: ['x=1', 'y=2', 'z=3'],
tags2: ['指标1', '指标2', '指标3'],
inputVisible: false,
inputVisible2: false,
inputValue: '',
inputValue2: '',
}
}
handleClose = (removedTag) => {
const tags = this.state.tags.filter(tag => tag !== removedTag);
console.log(tags);
this.setState({ tags });
}
showInput = () => {
this.setState({ inputVisible: true }, () => this.input.focus());
}
showInput2 = () => {
this.setState({ inputVisible2: true }, () => this.input.focus());
}
handleInputChange = (e) => {
this.setState({ inputValue: e.target.value });
}
handleInputChange2 = (e) => {
this.setState({ inputValue2: e.target.value });
}
handleInputConfirm = () => {
const state = this.state;
const inputValue = state.inputValue;
let tags = state.tags;
if (inputValue && tags.indexOf(inputValue) === -1) {
tags = [...tags, inputValue];
}
console.log(tags);
this.setState({
tags,
inputVisible: false,
inputValue: '',
});
}
handleInputConfirm2 = () => {
const state = this.state;
const inputValue2 = state.inputValue2;
let tags2 = state.tags2;
if (inputValue2 && tags2.indexOf(inputValue2) === -1) {
tags2 = [...tags2, inputValue2];
}
console.log(tags2);
this.setState({
tags2,
inputVisible2: false,
inputValue2: '',
});
}
saveInputRef = input => this.input = input
render() {
const uploadProps = {
name: 'file',
data: {"dataListId": ""},
action: '/services/v1/common/upload',
headers: {
"x-auth-token": localStorage.getItem('x-auth-token'),
},
onChange(info) {
if (info.file.status !== 'uploading') {
console.log(info.file, info.fileList);
}
if (info.file.status === 'done') {
message.success(`${info.file.name} file uploaded successfully`);
} else if (info.file.status === 'error') {
message.error(`${info.file.name} file upload failed.`);
}
},
};
const formItemLayout = {
labelCol: { span: 6 },
wrapperCol: { span: 16 },
};
let validate={
plugin:{
help:'',
status:'success'
},
label:{
help:'',
status:'success'
},
sourceField:{
help:'',
status:'success'
},
args:{
help:'',
status:'success'
}
};
const plugin=this.state.plugin;
const { tags,tags2, inputVisible, inputValue, inputVisible2, inputValue2 } = this.state;
return (
<div className="ant-layout-wrapper">
<div className="ant-layout-breadcrumb">
<Breadcrumb>
<Breadcrumb.Item>首页</Breadcrumb.Item>
<Breadcrumb.Item>机器学习配置</Breadcrumb.Item>
</Breadcrumb>
</div>
<div className="ant-layout-container">
<Form horizontal form={this.props.form}>
<FormItem required={true} {...formItemLayout} label="模型名称：" >
<Row>
<Col span={20}>
<Input type="text" name="label" placeholder="请输入模型名称"/>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'机器学习模型名称'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="学习框架：" help={validate.plugin.help} validateStatus={validate.plugin.status}>
<Row>
<Col span={20}>
<Select >
<Option value="Tensorflow">Tensorflow</Option>
<Option value="Caffe">Caffe</Option>
<Option value="Neuroph">Neuroph</Option>
</Select>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'支持的机器学习框架类型'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="算法参数：" style={{display:"on"}} help={validate.sourceField.help} validateStatus={validate.sourceField.status}>
<Row>
<Col span={20}>
<div>
{tags.map((tag, index) => {
const isLongTag = tag.length > 20;
const tagElem = (
<Tag key={tag} closable={index !== 0} afterClose={() => this.handleClose(tag)}>
{isLongTag ? `${tag.slice(0, 20)}...` : tag}
</Tag>
);
return isLongTag ? <Tooltip title={tag} key={tag}>{tagElem}</Tooltip> : tagElem;
})}
{inputVisible && (
<Input
ref={this.saveInputRef}
type="text"
size="small"
style={{ width: 78 }}
value={inputValue}
onChange={this.handleInputChange}
onBlur={this.handleInputConfirm}
onPressEnter={this.handleInputConfirm}
/>
)}
{!inputVisible && <Button size="small" type="dashed" onClick={this.showInput}>+参数</Button>}
</div>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'机器学习模型调用时需要的其它参数'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="特征指标：" style={{display:"on"}} help={validate.sourceField.help} validateStatus={validate.sourceField.status}>
<Row>
<Col span={20}>
<div>
{tags2.map((tag, index) => {
const isLongTag = tag.length > 20;
const tagElem = (
<Tag key={tag} closable={index !== 0} afterClose={() => this.handleClose(tag)}>
{isLongTag ? `${tag.slice(0, 20)}...` : tag}
</Tag>
);
return isLongTag ? <Tooltip title={tag} key={tag}>{tagElem}</Tooltip> : tagElem;
})}
{inputVisible2 && (
<Input
ref={this.saveInputRef}
type="text"
size="small"
style={{ width: 78 }}
value={inputValue2}
onChange={this.handleInputChange2}
onBlur={this.handleInputConfirm2}
onPressEnter={this.handleInputConfirm2}
/>
)}
{!inputVisible2 && <Button size="small" type="dashed" onClick={this.showInput2}>+特征</Button>}
</div>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'模型需要的特征指标描叙，模型计算依赖的数据'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="模型文件">
<Row>
<Col span={20}>
<Upload accept={'.zip'}>
<Button>
<Icon type="upload" /> 点击上传
</Button>
</Upload>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'机器学习训练后的文件'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="描叙信息" style={{display:"on"}} help={validate.args.help} validateStatus={validate.args.status}>
<Row>
<Col span={20}>
<Input.TextArea name="configJson" rows={4} placeholder="模型描叙信息。" />
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'模型描叙信息。'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
</Form>
</div>
</div>
);
}
}

55
webapp/src/component/datalist/modal/AddDataListRecord.jsx
View File

@ -1,5 +1,5 @@
import React from 'react';
import {Button,Checkbox,Select,Radio,Switch,Form,Row,Col,Icon,Modal,Input,InputNumber,Cascader,Tooltip } from 'antd';
import {Button,Checkbox,Select,Radio,Switch,Form,Row,Col,Icon,Modal,Input,InputNumber,Cascader,Tooltip,Upload, message } from 'antd';
const FormItem = Form.Item;
const RadioGroup = Radio.Group;
@ -15,13 +15,17 @@ export default class AddDataListRecord extends React.Component{
this.state={
visible:false,
importVisible: false,
dataRecord:'',
fieldNum:this.props.metaList.length
}
}
componentDidMount() {
console.log(this.props);
}
handleChange=(index,e)=>{
var value=e.target.value;
var valueArr=this.state.dataRecord.split(',');
@ -48,8 +52,15 @@ export default class AddDataListRecord extends React.Component{
showModal=()=>{
this.setState({
dataRecord:'',
visible:true
dataRecord: '',
visible: true
})
}
showModal2=()=>{
this.setState({
importVisible:true
})
}
@ -58,7 +69,7 @@ export default class AddDataListRecord extends React.Component{
param.dataListId=this.props.dataListId;
param.dataRecord=this.state.dataRecord;
FetchUtil('/datalistrecord/','PUT',JSON.stringify(param),
FetchUtil('/datalistrecord/','PUT', JSON.stringify(param),
(data) => {
this.setState({
visible:false
@ -73,15 +84,40 @@ export default class AddDataListRecord extends React.Component{
})
}
handleCancel2=()=>{
this.setState({
importVisible:false
})
}
render(){
const formItemLayout = {
labelCol: { span: 6 },
wrapperCol: { span: 16 },
};
const uploadProps = {
name: 'file',
data: {"dataListId": this.props.dataListId},
action: '/services/v1/datalistrecord/batchImportDataRecord',
headers: {
"x-auth-token": localStorage.getItem('x-auth-token'),
},
onChange(info) {
if (info.file.status !== 'uploading') {
console.log(info.file, info.fileList);
}
if (info.file.status === 'done') {
message.success(`${info.file.name} file uploaded successfully`);
} else if (info.file.status === 'error') {
message.error(`${info.file.name} file upload failed.`);
}
},
};
let valueArr=this.state.dataRecord.split(',');
return (
<span>
<Button onClick={this.showModal} type="primary">新增</Button>
<Button onClick={this.showModal} type="primary">新增</Button> &nbsp;&nbsp;
<Button onClick={this.showModal2} type="primary">导入数据</Button>
<span>&nbsp;&nbsp;</span> <a href="/res/dataRecord_temp.xlsx">下载数据模板</a>
<Modal title="新增记录" visible={this.state.visible} onOk={this.handleSubmit} onCancel={this.handleCancel}>
<Form horizontal form={this.props.form}>
{this.props.metaList.map(function(info,i){
@ -93,6 +129,13 @@ export default class AddDataListRecord extends React.Component{
}.bind(this))}
</Form>
</Modal>
<Modal title="导入数据" visible={this.state.importVisible} onCancel={this.handleCancel2} onOk={this.handleCancel2}>
<Upload {...uploadProps}>
<Button>
<Icon type="upload" /> Click to Upload
</Button>
</Upload>
</Modal>
</span>
);
}

7
webapp/src/component/model/Model.jsx
View File

@ -71,6 +71,8 @@ export default class Model extends React.Component{
case 'ruleList':
case 'historyRecordList':
key='activation';break;
case 'modelConfig':
key='modelConfig';break;
}
this.setState({
current:key
@ -186,6 +188,11 @@ export default class Model extends React.Component{
<Menu.Item key="abstractionList">
<Icon type="picture" />抽象处理
</Menu.Item>
<Menu.Item key="modelConfig">
<Icon type="setting" />机器学习配置
</Menu.Item>
<Menu.Item key="activation">
<Icon type="solution" />策略管理
</Menu.Item>

2
webapp/src/component/model/ModelList.jsx
View File

@ -187,7 +187,7 @@ export default class ModelList extends React.Component{
<div className="ant-layout-container">
<div className="ant-layout-content">
<div id="header">
<Form inline>
<Form layout="inline">
<FormItem label="模型名：">
<Input value={this.state.modelName} name="modelName" id="blue" onChange={this.handleChange}/>
</FormItem>

486
webapp/src/component/modelconfig/ModelConfig.jsx Normal file
View File

@ -0,0 +1,486 @@
import React from 'react';
import {Breadcrumb, Menu, Icon, Form, Upload, Input, Select, Card, Row, Col,Button, Tooltip, Tag, Modal, message} from 'antd';
const FormItem=Form.Item;
const Option = Select.Option;
import {FetchUtil} from '../utils/fetchUtil';
import {trim} from '../utils/validateUtil';
import EditModelConfParam from './modal/EditModelConfParam';
export default class ModelConfig extends React.Component{
constructor(props){
super(props);
this.state={
visible:false,
id : -1,
status:1,
args:'',
tags: ['argx=1', 'argy=2', 'argz=3'],
inputVisible: false,
inputValue: '',
model: null,
modelConfig: null,
name: "",
type: "TENSOR_DNN",
path: "",
comment:"",
tag: "",
operation: "",
absColumns: [],
selectCols: [],
fileList:[],
paramsList:[],
selectedKeys:[],
feed: "",
}
FetchUtil('/model/'+this.props.params.id,'GET','',
(data) => {
const model=data.data.model;
this.setState({
model:model
});
});
FetchUtil('/modelConfig/list/' + this.props.params.id,'GET','',
(data) => {
const modelConfig=data.data.modelConfig;
let paramVO = modelConfig.params[0];
let selectCols = [];
selectCols = paramVO.expressions.replace(/abstractions./g,"").split(",");
this.setState({
modelConfig: modelConfig,
id: modelConfig.id,
name: modelConfig.name,
type: modelConfig.type,
path: modelConfig.path,
comment: modelConfig.comment,
tag: modelConfig.tag,
operation: modelConfig.operation,
selectCols: selectCols,
paramsList: modelConfig.params,
fileList: [{
uid: -1,
name: modelConfig.path,
status: 'done',
}],
feed: paramVO.feed,
});
});
FetchUtil('/activation/absColumns/' + this.props.params.id,'GET','',
(data) => {
let absColumns =[];
let absDS= data.data.columns;
for (let i = 0; i < absDS.length; i++) {
absColumns.push(<Option key={absDS[i].value}>{absDS[i].label}</Option>);
}
//console.log(absColumns);
this.setState({
absColumns: absColumns,
});
});
}
handleClose = (removedTag) => {
const tags = this.state.tags.filter(tag => tag !== removedTag);
console.log(tags);
this.setState({ tags });
}
showInput = () => {
this.setState({ inputVisible: true }, () => this.input.focus());
}
// handleInputChange = (e) => {
// this.setState({ inputValue: e.target.value });
// }
handleInputConfirm = () => {
const state = this.state;
const inputValue = state.inputValue;
let tags = state.tags;
if (inputValue && tags.indexOf(inputValue) === -1) {
tags = [...tags, inputValue];
}
console.log(tags);
this.setState({
tags,
inputVisible: false,
inputValue: '',
});
}
handleChange = (e) => {
//console.log(`Selected: ${value}`);
//var state = this.state;
//state['selectedParams'] = trim(value);
let selectedKeys = e;
this.setState({selectedKeys});
}
handlInputChange=(e)=>{
var name = e.target.name;
var value = e.target.value;
var state = this.state;
state[name] = trim(value);
this.setState(state);
}
uploadHandleChange = (info) => {
let fileList = info.fileList;
fileList = fileList.slice(-1);
this.setState({ fileList });
}
handleSubmit = (isValidated, e) => {
e.preventDefault();
//console.log(e, isValidated);
if(!isValidated){
Modal.error({
title: '提交失败',
content: '请确认表单内容输入正确',
});
} else{
var param={};
var confParam = {};
param.id= this.state.id;
param.name= this.state.name;
param.type= this.state.type;
param.path= this.state.fileList.map(item => item.name).join();
param.comment= this.state.comment;
param.tag= this.state.tag;
param.operation= this.state.operation;
param.status= this.state.status;
param.modelId = this.state.model.id;
confParam.feed = this.state.feed;
confParam.expressions= this.state.selectedKeys.map(item=> "abstractions." + item).join();
param.confParam = confParam;
FetchUtil('/modelConfig/','PUT',JSON.stringify(param),
(data) => {
if(data.success){
message.success('修改成功');
}else{
message.error(data.msg);
}
this.setState({
visible:false
});
//this.props.reload();
});
}
}
saveInputRef = input => this.input = input
render() {
const plugin=this.state.plugin;
const { tags,path, inputVisible, inputValue, paramsList} = this.state;
let isValidated = true;
console.log("cols==", this.state.selectCols);
console.log("path==", path);
const uploadProps = {
name: 'file',
data: {"key": "machine"},
action: '/services/v1/common/upload',
headers: {
"x-auth-token": localStorage.getItem('x-auth-token'),
},
onChange: this.uploadHandleChange,
};
const formItemLayout = {
labelCol: { span: 6 },
wrapperCol: { span: 16 },
};
let validate={
name:{
help:'',
status:'success'
},
tag:{
help:'',
status:'success'
},
operation:{
help:'',
status:'success'
},
feed:{
help:'',
status:'success'
}
};
if(!this.state.name){
validate.name.help='请输入机器学习模型名称';
validate.name.status='warning';
isValidated=false;
}else {
let reg = /^[\u4e00-\u9fa5 \w]{2,20}$/;
let name = this.state.name;
if(!reg.test(name)){
validate.name.help='按照提示输入正确的名称';
validate.name.status='error';
isValidated=false;
}
}
if(!this.state.feed){
validate.feed.help='请输入参数名称';
validate.feed.status='warning';
isValidated=false;
}else {
let reg = /^[a-zA-z]\w{1,29}$/;
let feed = this.state.feed;
if(!reg.test(feed)){
validate.feed.help='按照提示输入feed名称';
validate.feed.status='error';
isValidated=false;
}
}
if(!this.state.tag){
validate.tag.help='请输入tag名称';
validate.tag.status='warning';
isValidated=false;
}else {
let reg = /^[a-zA-z]\w{1,29}$/;
let tag = this.state.tag;
if(!reg.test(tag)){
validate.tag.help='按照提示输入正确的名称';
validate.tag.status='error';
isValidated=false;
}
}
if(!this.state.operation){
validate.operation.help='请输入opration名称';
validate.operation.status='warning';
isValidated=false;
}else {
let reg = /^[a-zA-z]\w{1,29}\/?\w{1,29}$/;
let operation = this.state.operation;
if(!reg.test(operation)){
validate.operation.help='按照提示输入正确的名称';
validate.operation.status='error';
isValidated=false;
}
}
return (
<div className="ant-layout-wrapper">
<div className="ant-layout-breadcrumb">
<Breadcrumb>
<Breadcrumb.Item>机器学习配置</Breadcrumb.Item>
</Breadcrumb>
</div>
<div className="ant-layout-container">
<Form layout="horizontal" onSubmit={this.handleSubmit.bind(this, isValidated)}>
<FormItem required={true} {...formItemLayout} label="模型名称：" help={validate.name.help} validateStatus={validate.name.status} >
<Row>
<Col span={20}>
<Input type="text" name="name" value={this.state.name} placeholder="请输入模型名称" onChange={this.handlInputChange}/>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'机器学习模型名称'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="学习框架：" >
<Row>
<Col span={20} >
<Select value={this.state.type}>
<Option value="TENSOR_DNN">TENSOR_DNN</Option>
</Select>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'支持的机器学习框架类型'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="算法参数：" style={{display:"None"}} >
<Row>
<Col span={20}>
<div>
{tags.map((tag, index) => {
const isLongTag = tag.length > 20;
const tagElem = (
<Tag key={tag} closable={index !== 0} afterClose={() => this.handleClose(tag)}>
{isLongTag ? `${tag.slice(0, 20)}...` : tag}
</Tag>
);
return isLongTag ? <Tooltip title={tag} key={tag}>{tagElem}</Tooltip> : tagElem;
})}
{inputVisible && (
<Input
ref={this.saveInputRef}
type="text"
size="small"
style={{ width: 78 }}
value={inputValue}
onChange={this.handleInputChange}
onBlur={this.handleInputConfirm}
onPressEnter={this.handleInputConfirm}
/>
)}
{!inputVisible && <Button size="small" type="dashed" onClick={this.showInput}>+参数</Button>}
</div>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'机器学习模型调用时需要的其它参数'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="Tag：" help={validate.tag.help} validateStatus={validate.tag.status}>
<Row>
<Col span={20}>
<Input type="text" name="tag" value={this.state.tag} placeholder="tag" onChange={this.handlInputChange}/>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'tag'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="Operation：" help={validate.operation.help} validateStatus={validate.operation.status}>
<Row>
<Col span={20}>
<Input type="text" name="operation" value={this.state.operation} placeholder="operation" onChange={this.handlInputChange}/>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'Operation'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="模型文件">
<Row>
<Col span={20}>
<Upload {...uploadProps} accept={'.zip'} fileList={this.state.fileList}>
<Button>
<Icon type="upload" /> 点击上传
</Button>
</Upload>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'机器学习训练后的文件, 仅支持zip格式'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="描叙信息" style={{display:"on"}} >
<Row>
<Col span={20}>
<Input.TextArea name="comment" value={this.state.comment} rows={4} placeholder="模型描叙信息。" onChange={this.handlInputChange}/>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'模型描叙信息。'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="模型入参" help={validate.feed.help} validateStatus={validate.feed.status}>
{
paramsList.map((item,index) => {
const selected = item.expressions.replace(/abstractions./g,"").split(",");
console.log("selected:" ,selected);
let rows = (<Row key={index} id={item.id}>
<Col span={4}><Input type="text" name="feed" value={item.feed} placeholder="feed" /></Col>
<Col span={15} offset={1}>
<div>
<Select
mode="tags"
size={'default'}
placeholder="Please select"
value={selected}
style={{ width: '100%' }}
>
{this.state.absColumns}
</Select>
</div>
</Col>
<Col span={1} offset={1}>
<EditModelConfParam paramId={item.id} abstractions={this.state.absColumns} />
</Col>
</Row>);
return rows;
})
}
{
paramsList.length==0 ? (
<Row key={-1}>
<Col span={4}><Input type="text" name="feed" value={this.state.feed} placeholder="feed" onChange={this.handlInputChange}/></Col>
<Col span={15} offset={1}>
<div>
<Select
mode="tags"
size={'default'}
placeholder="Please select"
onChange={this.handleChange}
style={{ width: '100%' }}
>
{this.state.absColumns}
</Select>
</div>
</Col>
</Row>
)
: ""
}
</FormItem>
<FormItem>
<Row>
<Col span={20} offset={18}>
<Button type="primary" htmlType="submit">
更新配置
</Button>
</Col>
</Row>
</FormItem>
</Form>
</div>
</div>
);
}
}

212
webapp/src/component/modelconfig/modal/EditModelConfParam.jsx Normal file
View File

@ -0,0 +1,212 @@
import React from 'react';
import {Button,Checkbox,Select,Radio,Switch,Form,Row,Col,Icon,Modal,Input,InputNumber,Cascader,Tooltip,message } from 'antd';
const FormItem = Form.Item;
const RadioGroup = Radio.Group;
const Option = Select.Option;
import {FetchUtil} from '../../utils/fetchUtil';
import {trim} from '../../utils/validateUtil';
export default class EditModelConfParam extends React.Component{
constructor(props){
super(props);
this.state={
visible:false,
id :-1,
feed:'',
selectList:[],
indexed:false
}
FetchUtil('/modelConfigParam/' + this.props.paramId,'GET','',
(data) => {
let param = data.data.param;
this.setState({
id:param.id,
feed: param.feed,
selectList: param.expressions.replace(/abstractions./g,"").split(","),
})
});
// FetchUtil('/activation/absColumns/' + this.props.params.id,'GET','',
// (data) => {
// let absColumns =[];
// let absDS= data.data.columns;
// for (let i = 0; i < absDS.length; i++) {
// absColumns.push(<Option key={absDS[i].value}>{absDS[i].label}</Option>);
// }
// //console.log(absColumns);
// this.setState({
// absColumns: absColumns,
// });
// });
}
// 获取表格数据
// fetchData=()=>{
// FetchUtil('/modelConfigParam/'+ this.state.id,'GET',null,
// (data) => {
// const param=data.data.param;
// this.setState({
// id:param.id,
// feed: param.feed,
// expressions: param.expressions,
// });
// }
// )
// }
handleChange=(e)=>{
var name = e.target.name;
var value = e.target.value;
var state = this.state;
state[name] = trim(value);
this.setState(state);
}
handleSelect=(value)=>{
this.setState({selectList: value});
}
showModal=()=>{
//this.fetchData();
this.setState({
visible:true
})
}
onCheck=(e)=>{
if(e.target.checked && this.props.indexedAll>=8){
Modal.warning({
title: '提示信息',
content: '索引已超过8项！',
});
}else {
this.setState({
indexed:e.target.checked
});
}
};
handleSubmit=(validated)=>{
if(!validated){
Modal.error({
title: '提交失败',
content: '请确认表单内容输入正确',
});
}
else{
var param = {};
param.id = this.state.id;
param.feed = this.state.feed;
param.expressions = this.state.selectList.map(item => 'abstractions.' + item).join();
FetchUtil('/modelConfigParam/', 'PUT', JSON.stringify(param),
(data) => {
if (data.success) {
message.success('修改成功');
} else {
message.error(data.msg);
}
this.setState({
visible: false
});
this.props.reload();
});
}
}
handleCancel=()=>{
this.setState({
visible:false
})
}
render(){
const formItemLayout = {
labelCol: { span: 6 },
wrapperCol: { span: 16 },
};
let validate={
feed:{
help:'',
status:'success'
},
label:{
help:'',
status:'success'
},
fieldType:{
help:'',
status:'success'
}
};
let isValidated=true;
if(!this.state.feed){
validate.feed.help='请输入参数名称';
validate.feed.status='warning';
isValidated=false;
}else {
let reg = /^[a-zA-z]\w{2,29}$/;
let feed = this.state.feed;
if(!reg.test(feed)){
validate.feed.help='按照提示输入正确的名称';
validate.feed.status='error';
isValidated=false;
}
}
return (
<span>
<Tooltip title="编辑" onClick={this.showModal}><a>编辑</a></Tooltip>
<Modal title="编辑参数" visible={this.state.visible} onOk={this.handleSubmit.bind(this, isValidated)} onCancel={this.handleCancel}>
<Form layout="horizontal" form={this.props.form}>
<FormItem required={true} {...formItemLayout} label="feed：" help={validate.feed.help} validateStatus={validate.feed.status}>
<Row>
<Col span={20}>
<Input type="text" name="feed" value={this.state.feed} onChange={this.handleChange} />
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'2-30位英文字母、数字、下划线的组合，以英文字母开头：xyz001'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
<FormItem required={true} {...formItemLayout} label="特征指标：">
<Row>
<Col span={10}>
<Select
mode="tags"
size={'default'}
placeholder="Please select"
value={this.state.selectList}
onChange={this.handleSelect}
style={{ width: '100%' }}
>
{this.props.abstractions}
</Select>
</Col>
<Col span={2} offset={1}>
<Tooltip placement="right" title={'选择模型需要的特征指标'}>
<Icon style={{fontSize:16}} type="question-circle-o" />
</Tooltip>
</Col>
</Row>
</FormItem>
</Form>
</Modal>
</span>
);
}
}

8
webapp/src/component/preItem/modal/AddPreItem.jsx
View File

@ -24,7 +24,7 @@ export default class AddPreItem extends React.Component{
status:1,
args:'',
reqType:'GET',
configJson:''
configJson:JSON.stringify({})
}
}
@ -46,12 +46,12 @@ export default class AddPreItem extends React.Component{
state['sourceLabel']='';
state['args']='';
state['status']=1;
state['configJson']='';
state['configJson']= JSON.stringify({});
}
state[name] = trim(value);
if(name=='sourceField'){
state['sourceLabel']=this.props.fieldList.filter(x=>x.fieldName==value)[0].label;
state['sourceLabel']= this.props.fieldList.filter(x => x.fieldName==value)[0].label;
}
this.setState(state);
@ -83,7 +83,7 @@ export default class AddPreItem extends React.Component{
args:'',
reqType:'GET',
visible:true,
configJson:''
configJson:JSON.stringify({})
})
}

4
webapp/src/component/preItem/modal/EditPreItem.jsx
View File

@ -24,7 +24,7 @@ export default class EditPreItem extends React.Component{
status:1,
args:'',
reqType:'GET',
configJson:'',
configJson:JSON.stringify({}),
preItem:null
}
@ -64,7 +64,7 @@ export default class EditPreItem extends React.Component{
state['sourceField']='';
state['sourceLabel']='';
state['args']='';
state['configJson']='';
state['configJson']=JSON.stringify({});
}
state[name] = trim(value);

4
webapp/src/component/report/ListEvent.jsx
View File

@ -110,9 +110,7 @@ export default class ListEvent extends React.Component{
FetchUtil('/event/search','POST',JSON.stringify(param),
(data) => {
this.setState({
tData:data.data.page.list.map((info)=>{
info=info.replace(/":(-?\d+)/g, "\":\"$1\"");
return JSON.parse(info)}),
tData:data.data.page.list,
pageNo:data.data.page.pageNum
});
if(data.data.page.rowCount > 9990){

11
webapp/src/index.jsx
View File

@ -20,7 +20,8 @@ import RuleGraph from './component/report/RuleGraph';
import ListRule from './component/report/ListRule';
import Rule from './component/report/ListRule';
import DashBoard from './component/report/DashBoard';
import ConfigCenter from './component/config/ConfigCenter';
import ModelConfig from './component/modelconfig/ModelConfig';
//import Test from './component/test/Test';
@ -55,7 +56,7 @@ class Welcome extends React.Component{
<div>
<div className="ibox">
<div className="ibox-content">
<h2>欢迎登录反欺诈系统管理平台！</h2>
<h2>欢迎登录风控引擎管理平台！</h2>
</div>
</div>
</div>
@ -96,6 +97,7 @@ class AppRoutes extends React.Component{
<Route path="/ruleList/:id/:activationId" component={RuleList}/>
<Route path="/historyRecordList/:id/:activationId/:ruleId" component={HistoryRecordList}/>
<Route path="/abstractionList/:id" component={AbstractionList}/>
<Route path="/modelConfig/:id" component={ModelConfig}/>
{/*<Route path="/test/:id" component={Test}/>*/}
</Route>
@ -108,7 +110,10 @@ class AppRoutes extends React.Component{
<Route path="/ruleid/:modelId/:ruleId/:activationName" component={ListEvent}/>
<Route path="/dashboard" component={DashBoard}/>
</Route>
</Route>
<Route path="/config" component={ConfigCenter}>
</Route>
</Route>
<Route path="*" component={Index}>
<IndexRoute component={NotFound} />

26
webapp/webpack.config.dev.js
View File

@ -16,26 +16,26 @@ const devWebpackConfig = merge(baseWebpackConfig, {
},
devServer: {
contentBase: path.join(__dirname, 'dist'),
port: 8080,
port: 3000,
disableHostCheck: true,
host: '0.0.0.0',
host: 'localhost',
compress: true,
inline: true,
hot: true,
overlay: true,
disableHostCheck: true, // 新增该配置项
proxy: [{
context: ["/services/v1/"],
target: "http://10.50.3.218:8080",
changeOrigin: true,
secure: false,
onProxyRes: function (proxyRes, req, res) { //
// console.log(proxyRes)
let proxyHost = proxyRes.req.getHeader('host');
let proxyPath = proxyRes.req.path;
//console.log(host, path)
console.log(`Proxy ${req.get('host')}${req.path} -> ${proxyHost}${proxyPath}`)
}
context: ["/services/v1/"],
target: "http://localhost:8080",
changeOrigin: true,
secure: false,
onProxyRes: function(proxyRes, req, res) { //
// console.log(proxyRes)
let proxyHost = proxyRes.req.getHeader('host');
let proxyPath = proxyRes.req.path;
//console.log(host, path)
console.log(`Proxy ${req.get('host')}${req.path} -> ${proxyHost}${proxyPath}`)
}
}]
},
plugins: [