知学云性能优化规范¶
本规范根据产品中心性能压测过程中发现的问题进行优化经验累积，结合多次性能压测爆发的问题整理出一套性能优化编码规范，为知学云开发人员提供规范与指引。
修订记录¶
| 提交者 | 更新日期 | 备注 |
|---|---|---|
| 架构部 | 2020.6.8 | 创建 |
| 架构部 | 2020.6.12 | 初稿 |
代码优化规范¶
-
【强制】多次循环完成的事情，优化为一次性循环完成。
反例：
private Map<String,List<QuestionCopy>> createInsertQuestionCopys( List<QuestionCopy> questionCopies, List<PaperClassQuestion> paperClassQuestions, String examId) { Map<String, QuestionCopy> questionIdMap = questionCopies.stream() .collect(Collectors.toMap(QuestionCopy::getQuestionId, e -> e, (k, v) -> v)); List<QuestionCopy> temp = paperClassQuestions.stream().map(q -> { QuestionCopy questionCopy = new QuestionCopy(); questionCopy.forInsert(); questionCopy.setQuestionId(q.getQuestionId()); questionCopy.setExamId(examId); questionCopy.setScore(Integer.valueOf(q.getScore())); questionCopy.setSequence(q.getSequence()); return questionCopy; }).collect(Collectors.toList()); List<QuestionCopy> needInserts = temp.stream().filter(t -> { return questionIdMap.get(t.getQuestionId()) == null; }).collect(Collectors.toList()); (1) List<QuestionCopy> noNeedInserts = temp.stream().filter(t -> { return questionIdMap.get(t.getQuestionId()) != null; }).collect(Collectors.toList()); (2) noNeedInserts = noNeedInserts.stream().map(q -> { QuestionCopy questionCopy = questionIdMap.get(q.getQuestionId()); return questionCopy; }).collect(Collectors.toList()); (3) Map<String,List<QuestionCopy>> result = new HashMap<>(); result.put("needInsert", needInserts); noNeedInserts.addAll(needInserts); result.put("paperQuestions", noNeedInserts); return result; }正例：
private Map<String,List<QuestionCopy>> createInsertQuestionCopys( List<QuestionCopy> questionCopies, List<PaperClassQuestion> paperClassQuestions, String examId) { Map<String, QuestionCopy> questionIdMap = questionCopies.stream() .collect(Collectors.toMap(QuestionCopy::getQuestionId, e -> e, (k, v) -> v)); List<QuestionCopy> noNeedInserts = new ArrayList<>(); List<QuestionCopy> needInserts = new ArrayList<>(); paperClassQuestions.forEach(q -> { QuestionCopy qc = questionIdMap.get(q.getQuestionId()); if(qc != null){ noNeedInserts.add(qc); return; } QuestionCopy questionCopy = new QuestionCopy(); questionCopy.forInsert(); questionCopy.setQuestionId(q.getQuestionId()); questionCopy.setExamId(examId); questionCopy.setScore(q.getScore()); questionCopy.setSequence(q.getSequence()); needInserts.add(questionCopy); }); noNeedInserts.addAll(needInserts); Map<String,List<QuestionCopy>> result = new HashMap<>(); result.put("needInsert", needInserts); noNeedInserts.addAll(needInserts); result.put("paperQuestions", noNeedInserts); return result; } -
【强制】循环n次计算，优化为一次计算
反例：
正例：public List<Member> find(String name, String organizationId) { 省略... List<Organization> companyList = findCompanys(organizationId); // 填充member对象的归属机构 members.forEach(m -> { m.setCompOrganization(companyList.stream().filter(c -> c.getId().equals(m.getCompanyId())).findAny().orElse(new Organization())); }); 省略... }public List<Member> find(String name, String organizationId) { 省略... Map<String, Organization> companyMap = findCompanyMap(organizationId); // 填充member对象的归属机构 members.forEach(m -> { m.setCompOrganization(companyMap.get(m.getCompanyId())); }); 省略... } -
【强制】一次性处理大量数据，优化为分批处理
反例：
正例：private void handleExportTaskMsg(Message msg) { 省略... int memberCount = 1; // 一次查询所有数据，rpc超时，内存溢出问题 PagedResult<Member> pagedResult = memberService.find(1, Integer.MAX_VALUE, grantOrganizationIds, name, fullName, phoneNumber, email, jobId, credentialValue, incumbencyStatus, status, from, Optional.empty(), orgId, Optional.empty()); List<Member> memberList = pagedResult.getItems(); 省略... ｝private void handleExportTaskMsg(Message msg) { 省略... // 采用do{} while() 分批获取 do { pagedResult = findExortData((p) -> { PagedResult<Member> rst = null; try { rst = find(page.get(), DEF_PAGE_SIZE, grantOrganizationMap, name, fullName, phoneNumber, email, jobId, credentialValue, incumbencyStatus, status, memberType, Optional.empty(), orgId, positionId); }catch(Exception ex) { String err = "memberService.find failed. page=" + page.get(); LOGGER.error(err, ex); } return rst; }, DEF_RETRY_COUNT, DEF_RETRY_INTERVAL); if (pagedResult == null) { findFailed = true; break; } ... } while(page.get() < 10000 && pagedResult.getItems() != null && pagedResult.getItems().size() == DEF_PAGE_SIZE); 省略... } -
【强制】大事物方法采用MQ异步消息拆分完成
反例：
private void doAnswerRecordCounting(Message message) { 省略... /** * 一个事物里面需要做的事情,事物释放时间太长，并发下易锁表 * 算分预处理： * 1)删除非本试卷的答题记录 * 2)删除阅读理解母题答题记录 * 3)更新答题次数 * 4)更新任务状态 * 5)发送计算分数消息 */ transactionTemplate.execute(new TransactionCallbackWithoutResult() { @Override protected void doInTransactionWithoutResult(TransactionStatus status) { if (answerRecordList != null) { ExamRecord examRecord = getExamRecord(examRecordId); if(examRecord == null) { LOGGER.error("exam/AnswerRecordListener, examRecord is null，examRecordId:{}", examRecordId); return; } //通过redis的全量试题来补偿真正的已答试题记录(生产环境有可能增量提交时候造成丢失答题记录) //删除多余的答题记录 List<AnswerRecord> afterRemoveExtra = deleteExtraAnswers(answerRecordList, examRecord); updateQuestionAnswerNum(afterRemoveExtra); List<AnswerRecord> updateList = new ArrayList<>(); //计算客观题分数 afterRemoveExtra.forEach(a -> { 省略... updateList.add(a); } }); //批量更新 batchUpdateAnswerRecord(updateList); //批量删除 deleteReadingAnswer(shouldDeleteAnswerIds); //判断是否需要评卷 boolean noNeedMark = getNoNeedMark(afterRemoveExtra, updateList); Exam exam = getExamFromCacheOrDB(examRecord.getExamId()); if (exam != null) { //得出考试记录结果 examRecord.setAnsweredCount(finalCalculateAnsweredCount(afterRemoveExtra, exam, examRecord)); examRecord.setNoAnswerCount(getNoAnswerCount(getQuestionNum(exam.getPaperClass()), examRecord.getAnsweredCount())); List<AnswerRecord> realAnswerRecords = updateList .stream() .filter(ar -> null != ar.getId() && !shouldDeleteAnswerIds.contains(ar.getId())) .collect(Collectors.toList()); confirmExamRecord(examRecord, exam, noNeedMark, realAnswerRecords); LOGGER.info("exam/AnswerRecordListener, 得出考试记录结果======end"); //创建待办 if (!noNeedMark) { messageSender.send(MessageTypeContent.CREATE_TO_DO, MessageHeaderContent.ID, examRecord.getId()); } //更新任务 messageSender.send(MessageTypeContent.AFTER_SUBMIT_PAPER, ExamRecord.SUBMIT_PAPER_EXAM_RECORD_ID, examRecord.getId()); } } } }); }正例：
private void preAnswerRecordCounting(Message message) { 省略... // 将更新任务和算分发送mq异步处理 transactionTemplate.execute(new TransactionCallbackWithoutResult() { @Override protected void doInTransactionWithoutResult(TransactionStatus status) { // 删除多余的答题记录 List<AnswerRecord> afterRemoveExtra = deleteExtraAnswers(answerRecordList, examRecord); // 更新答题次数 updateQuestionAnswerNum(afterRemoveExtra); } }); // 更新任务 messageSender.send( MessageTypeContent.AFTER_SUBMIT_PAPER, ExamRecord.SUBMIT_PAPER_EXAM_RECORD_ID, examRecordId); // 发送算分消息 messageSender.send( MessageTypeContent.EXAM_ANSWER_RECORD_UPDATE_COUNT, MessageHeaderContent.ID, examRecordId); } -
【强制】针对mongodb并发读写，分开同一时刻读写同一个document，如果业务强制需要可以采用缓存读的方式挫开同一时刻并发写，来防止读写锁竞争
反例：
正例：/** * 验证当前登录人数是否超过了最大在线数 * @param company */ private Integer validateMaxOnlineNumber(Company company) { Integer maxOnlineNumber = company.getMaxOnlines(); if (null==maxOnlineNumber || maxOnlineNumber==0){ return maxOnlineNumber; } //获取当前最大在线人数，这里登录过程会写mongodb在线登录记录，并发情况下同一时刻读写会产生mongodb读写锁竞争，导致cpu飙升 int crrentOnlineNumber = onlineInfoService.getOnlineCount(company.getId(), Optional.empty(), Optional.of(true)); ErrorCode.LimitMaxOnlineNumber.throwIf((maxOnlineNumber - crrentOnlineNumber) < 0); return crrentOnlineNumber; }/** * 验证当前登录人数是否超过了最大在线数 * @param company */ private Integer validateMaxOnlineNumber(Company company) { Integer maxOnlineNumber = company.getMaxOnlines(); if (null==maxOnlineNumber || maxOnlineNumber==0){ return maxOnlineNumber; } //获取当前最大在线人数，加入五秒缓存，挫开并发场景下读写锁竞争 int crrentOnlineNumber = cache.get(ONLINE_COUNT_KEY + "#" + company.getId(), () -> onlineInfoService.getOnlineCount(company.getId(), Optional.empty(), Optional.of(true)), ONLINE_COUNT_CACHE_SECONDS); ErrorCode.LimitMaxOnlineNumber.throwIf((maxOnlineNumber - crrentOnlineNumber) < 0); return crrentOnlineNumber; } -
【强制】在同一个页面区域前端发起一个请求获取的数据在多个地方使用，不应该多次发起重复请求
-
【推荐】针对实时性要求不高的数据进行缓存优化，如排行榜，最新，最热列表数据
-
【推荐】可以异步计算的业务数据可以采用MQ消息队列优化，如浏览数，个人总积分，个人总学习时长
-
【推荐】针对请求返回数据量小的请求，进行请求合并，减少网络开销，提高并发吞吐量
SQL优化规范¶
-
【强制】禁止使用select * 返回所有字段,在返回的数据量比较大的情况下，mysql无法走内存缓存就会走io磁盘，导致资源占用飙升
反例：
e.select(Fields.start() .add(MEMBER.fields()) .add(ORGANIZATION.fields()) .end()).from(MEMBER) .leftJoin(MEMBER_DETAIL) .on(MEMBER.ID.eq(MEMBER_DETAIL.MEMBER_ID)) .innerJoin(ORGANIZATION) .on(MEMBER.ORGANIZATION_ID.eq(ORGANIZATION.ID)) .where(conditions) .orderBy(MEMBER.CREATE_TIME.desc()).limit((page - 1) * pageSize, pageSize)正例：
e.select(Fields.start() .add(MEMBER.ID) .add(MEMBER.NAME) .add(MEMBER.FULL_NAME) .add(MEMBER.INIT) .add(MEMBER.SEX) .add(MEMBER.ORGANIZATION_ID) .add(MEMBER.MAJOR_POSITION_ID) .add(MEMBER.STATUS) .add(MEMBER_DETAIL.HEAD_PORTRAIT) .add(MEMBER_DETAIL.HEAD_PORTRAIT_PATH) .add(ORGANIZATION.ID) .add(ORGANIZATION.NAME) .add(ORGANIZATION.COMPANY_ID) .add(ORGANIZATION.LEVEL) .end()).from(MEMBER) .leftJoin(MEMBER_DETAIL) .on(MEMBER.ID.eq(MEMBER_DETAIL.MEMBER_ID)) .innerJoin(ORGANIZATION) .on(MEMBER.ORGANIZATION_ID.eq(ORGANIZATION.ID)) .where(conditions) .orderBy(MEMBER.CREATE_TIME.desc()).limit((page - 1) * pageSize, pageSize) -
【强制】前期设计上尽量规避使用distinct做数据去重,如果规避不掉数据量很小推荐采用in查询代替或者java内存去重
反例：
正例：// 返回distinct过滤个人授权节点 List<Organization> organizationList = grantItemDao.execute(e -> e.selectDistinct(GRANT_ITEM.ORGANIZATION_ID, ORGANIZATION.PATH, GRANT_ITEM.CHILD_FIND) .from(GRANT_ITEM) .leftJoin(GRANT_MEMBER_ITEM).on(GRANT_ITEM.ID.eq(GRANT_MEMBER_ITEM.ITEM_ID)) .innerJoin(ORGANIZATION).on(GRANT_ITEM.ORGANIZATION_ID.eq(ORGANIZATION.ID)) .leftJoin(ROLE_MENU).on(GRANT_ITEM.ROLE_ID.eq(ROLE_MENU.ROLE_ID)) .leftJoin(MENU).on(ROLE_MENU.MENU_ID.eq(MENU.ID)) .where(MENU.URI.eq(uri).and(GRANT_MEMBER_ITEM.MEMBER_ID.eq(memberId))) .fetch(r -> { Organization organization = new Organization(); organization.setId(r.getValue(GRANT_ITEM.ORGANIZATION_ID)); organization.setPath(r.getValue(ORGANIZATION.PATH)); organization.setChildFind(String.valueOf(r.getValue(GRANT_ITEM.CHILD_FIND))); return organization; }));// 这里个人的授权节点并不多，可以采用java内存去重 List<Organization> organizationList = grantItemDao.execute(e -> e.select(GRANT_ITEM.ORGANIZATION_ID, ORGANIZATION.PATH, GRANT_ITEM.CHILD_FIND) .from(GRANT_ITEM) .leftJoin(GRANT_MEMBER_ITEM).on(GRANT_ITEM.ID.eq(GRANT_MEMBER_ITEM.ITEM_ID)) .innerJoin(ORGANIZATION).on(GRANT_ITEM.ORGANIZATION_ID.eq(ORGANIZATION.ID)) .leftJoin(ROLE_MENU).on(GRANT_ITEM.ROLE_ID.eq(ROLE_MENU.ROLE_ID)) .leftJoin(MENU).on(ROLE_MENU.MENU_ID.eq(MENU.ID)) .where(MENU.URI.eq(uri).and(GRANT_MEMBER_ITEM.MEMBER_ID.eq(memberId))) .fetch(r -> { Organization organization = new Organization(); organization.setId(r.getValue(GRANT_ITEM.ORGANIZATION_ID)); organization.setPath(r.getValue(ORGANIZATION.PATH)); organization.setChildFind(String.valueOf(r.getValue(GRANT_ITEM.CHILD_FIND))); return organization; })); // 转化为set去重 Set<Organization> organizationSet = organizationList.stream().collect(Collectors.toSet()); -
【强制】前期设计上尽量规避 order by 排序字段超过两个以上,如果数据量不大可以考虑内存排序
-
【强制】联表查询不要超过三张表，超过了想办法拆分sql
反例：
// 组织名称，岗位名称，职位名称这些不参与查询，可以拆分sql处理 SelectSeekStep1<Record, Long> selectStep = querySelect.from(MEMBER_SUPERIOR) .leftJoin(MEMBER).on(MEMBER.ID.eq(MEMBER_SUPERIOR.MEMBER_ID)) .leftJoin(MEMBER_DETAIL).on(MEMBER_DETAIL.MEMBER_ID.eq(MEMBER.ID)) .leftJoin(ORGANIZATION).on(ORGANIZATION.ID.eq(MEMBER.ORGANIZATION_ID)) .leftJoin(POSITION).on(POSITION.ID.eq(MEMBER.MAJOR_POSITION_ID)) .leftJoin(JOB).on(JOB.ID.eq(MEMBER.JOB_ID)) .where(conditions) .orderBy(MEMBER.CREATE_TIME.desc());正例：
SelectSeekStep1<Record, Long> selectStep = querySelect.from(MEMBER_SUPERIOR) .leftJoin(MEMBER).on(MEMBER.ID.eq(MEMBER_SUPERIOR.MEMBER_ID)) .leftJoin(MEMBER_DETAIL).on(MEMBER_DETAIL.MEMBER_ID.eq(MEMBER.ID)) .where(conditions).orderBy(MEMBER.CREATE_TIME.desc()); // 由于分页控制了数量大小，组织，岗位，职位拆分处理 Map<String, Organization> organizationMap = organizationIdSet.isEmpty() ? new HashMap<>() : organizationDao.execute(e -> e.select(ORGANIZATION.ID, ORGANIZATION.NAME).from(ORGANIZATION) .where(ORGANIZATION.ID.in(organizationIdSet)) .fetchInto(Organization.class).stream().filter(o -> o.getId() != null) .collect(Collectors.toMap(Organization::getId, o -> o))); Map<String, Position> positionMap = positionIdSet.isEmpty() ? new HashMap<>() : positionDao.execute(e -> e.select(POSITION.ID, POSITION.NAME).from(POSITION) .where(POSITION.ID.in(positionIdSet)) .fetchInto(Position.class).stream().filter(p -> p.getId() != null) .collect(Collectors.toMap(Position::getId, p -> p))); Map<String, Job> jobMap = jobIdSet.isEmpty() ? new HashMap<>() : jobDao.execute(e -> e.select(JOB.ID, JOB.NAME).from(JOB) .where(JOB.ID.in(jobIdSet)) .fetchInto(Job.class).stream().filter(j -> j.getId() != null) .collect(Collectors.toMap(Job::getId, j -> j))); -
【强制】数据超过20万以上的表不要使用sum，count，group by这些统计分组函数
-
【强制】in操作集合的大小控制在100～200以内 反例：
正例：// in数据量过大导致慢查 answerRecordDao.delete(deletes.stream() .map(AnswerRecord::getId) .collect(Collectors.toList()));// 控制in size大小，分批处理 PartitionUtil.listPartition(deletes, deleteList -> answerRecordDao.delete(deleteList.stream() .map(AnswerRecord::getId) .collect(Collectors.toList())), DELETE_SIZE);
业务优化规范¶
-
【推荐】count分页显示在数据量大的情况下，优化手段有限，推荐采用下拉加载分页或者点击加载更多的设计交互
-
【推荐】针对大批量数据人导入，导出操作，设计上推荐采用异步交互方式，体验上更好
-
【推荐】针对排行榜，统计型数据显示，推荐采用T＋1延时方案设计
-
【推荐】针对排序规则比较复杂的业务数据，推荐采用分解动作让用户点击某个元素排序，既可以满足用户选择性诉求，也可以满足程序设计上的性能
-
【推荐】针对数据来自各个业务模块综合显示的需求，推荐设计上尽量提前布局设计，不能依靠程序收集排序显示，这样性能优化有限，如活动首页，个人任务这些场景就目前程序设计性能优化空间不大
-
【推荐】针对折叠多层显示结构的数据，数据量比较大的情况设计上可以分层点解按需加载，如课程多级目录，组织管理树
部署优化规范¶
- 待续...